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Executive  Summary 


Problem 

Traditional  Behavioral  Task  Analysis  techniques  used  by  Navy  Instructional  Systems 
Specialists  (ISSs)  to  design  and  revise  courses  provide  little  guidance  in  eliciting  cognitive 
information.  The  Navy  Personnel  Research  and  Development  Center  recognized  a  need  to 
develop  methodologies  that  would  enable  ISSs  to  systematically  incorporate  more  cognitive 
information  into  Navy  courses.  Existing  Cognitive  Task  Analysis  techniques  tend  to  be  labor  and 
resource  intensive,  require  extensive  training  to  conduct,  and  are  usually  not  feasible  for  use  in  an 
applied  setting.  The  need  for  an  Applied  Cognitive  Task  Analysis  (ACTA)  methodology  to  be 
used  by  Navy  ISSs  and  other  Instructional  Designers  in  designing  and  revising  courses  drove  this 
project. 

Objective 

The  primary  objectives  of  this  project  were  as  follows: 

•  To  develop  a  set  of  ACTA  tools  that  elicit  important  cognitive  aspects  of  expert 
performance,  and  that  are  easier  to  learn  than  earlier  Cognitive  Task  Analysis  Methods. 

•  To  conduct  an  evaluation  study  investigating  the  usability,  validity,  and  reliability  of  the 
ACTA  tools. 

•  To  develop  a  stand-alone  instructional  package  to  train  ISSs  and  private  sector 
Instructional  Designers  to  use  the  ACTA  tools. 

•  To  prepare  for  the  commercialization  of  the  ACTA  tools. 

Approach 

Our  approach  was  to  work  from  our  own  experience  conducting  Cognitive  Task  Analysis  and 
the  experiences  of  other  researchers  as  described  in  the  literature  to  formulate  candidate 
methodologies.  We  then  conducted  a  user  analysis,  interviewing  Navy  ISSs,  Navy  instmctors, 
and  private  sector  Instructional  Designers.  The  goal  of  this  user  analysis  was  to  understand  the 
methods  currently  used  by  Navy  ISSs  and  private  sector  Instructional  Designers,  the  challenges 
these  training  professionals  face,  and  the  environment  in  which  these  training  professionals 
operate.  Findings  from  the  user  analysis  guided  revisions  and  refinements  of  the  candidate 
methods.  Subsequently,  workshops  were  conducted  for  both  Navy  and  private  sector  audiences 
to  gain  feedback  from  potential  users  of  the  ACTA  tools.  The  tools  were  continually  refined 
based  on  this  feedback. 
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Once  a  strong  set  of  ACTA  tools  had  been  established,  an  evaluation  was  conducted  to 
investigate  the  usability,  validity,  and  reliability  of  the  methods.  In  the  final  portion  of  the 
project,  a  stand-alone  instructional  package  in  the  form  of  the  multimedia  software  was 
developed  to  teach  Navy  ISSs  and  private  sector  Instructional  Designers  to  use  the  ACTA  tools. 

Conclusions 

The  goal  of  this  project  was  to  begin  to  move  Cognitive  Task  Analysis  out  of  the  research 
community  into  applied  communities.  We  have  made  important  progress  in  this  direction.  We 
have  developed  streamlined  methods  of  Cognitive  Task  Analysis.  Our  evaluation  study  indicates 
that  the  methods  are  usable  and  aid  in  the  development  of  important,  accurate  training  materials 
addressing  cognitive  issues.  In  addition,  we  have  developed  a  CD-based  stand-alone  instructional 
package  which  will  make  the  ACTA  tools  widely  accessible.  A  review  of  the  software  conducted 
with  both  Navy  ISSs  and  private  sector  Instructional  Designers  indicates  that  the  software  is 
successful  in  communicating  the  concepts  behind  the  ACTA  techniques  and  the  procedure  for 
conducting  each  technique. 

Recommendations 

There  are  several  areas  in  which  we  would  recommend  future  research  to  expand  on  what  has 
been  accomplished  with  the  current  ACTA  project.  First,  there  is  a  need  for  team  Cognitive  Task 
Analysis  methods.  Although  team  issues  are  not  a  factor  in  all  cognitively  complex  tasks,  there 
are  certainly  many  situations  in  which  team  issues  are  critical.  Methods  aimed  at  better 
understanding  information  flow  among  team  members,  roles  and  functions  of  team  members,  etc. 
are  needed.  Secondly,  the  ACTA  techniques  could  provide  valuable  information  for  systems  and 
interface  design.  Thus  far  our  efforts  have  focused  on  applying  ACTA  to  training  design  and 
course  revision.  Future  research  could  adapt  the  ACTA  techniques  for  use  by  system  and 
interface  designers. 

A  third  recommendation  for  future  research  is  to  examine  the  means  by  which  training  should 
be  administered  to  teach  cognitive  skills.  What  types  of  cognitive  skills  are  best  taught  in  the 
classroom  vs.  in  an  actual  work  setting?  Are  new  training  techniques  required  to  sufficiently 
address  the  cognitive  demands  of  today’s  workplace?  In  addition,  future  research  should 
investigate  whether  effective  knowledge  elicitation  and  representation  tools  can  be  developed 
that  reduce  the  role  of  the  interviewer  in  this  process.  Can  portions  of  the  Cognitive  Task 
Analysis  be  automated?  Is  it  possible  for  a  Subject  Matter  Expert  (SME)  to  conduct  a  Cognitive 
Task  Analysis  on  him/herself?  A  final  direction  for  future  research  is  to  identify  better  metrics  to 
assess  the  usability,  validity,  and  reliability  of  Cognitive  Task  Analysis  tools,  so  that  different 
methodologies  can  be  compared. 
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Introduction 


The  impact  of  technology  on  many  tasks  and  functions  has  resulted  in  greatly  increased 
demands  on  the  cognitive  skills  of  workers.  Howell  and  Cooke  (1989)  have  argued  that  with 
advances  in  technology  and  machine  responsibility,  we  have  increased  (rather  than  lowered) 
cognitive  demands  on  humans.  More  procedural  or  predictable  tasks  are  now  handled  by  smart 
machines,  while  humans  have  become  responsible  for  tasks  that  require  inference,  diagnoses, 
judgment,  and  decision  making. 

The  increase  in  the  cognitive  demand  placed  on  workers  has  created  a  need  for  training  that 
targets  cognitive  skills.  In  most  cases,  however,  the  task  analyses  that  drive  training  development 
are  conducted  using  methodologies  that  focus  primarily  on  behaviors.  Behavioral  task  analytic 
techniques,  which  have  proven  so  effective  for  cataloging  the  functional  and  procedural  aspects  of 
tasks,  are  often  not  adequate  for  understanding  cognitively-complex  tasks.  The  training  community 
needs  tools  that  will  allow  access  to  experience-based  cognitive  skills.  Without  such  tools,  more  and 
more  of  the  critical  job  elements  will  be  missed  altogether — neither  adequately  identified  nor 
documented  and  therefore  not  included  in  training  programs. 

The  primary  goal  of  this  project  was  to  develop  streamlined  methods  of  Cognitive  Task 
Analysis  that  would  fill  this  need.  The  intent  was  to  provide  tools  that  would  allow  Instructional 
Designers  to  capture  cognitive  as  well  as  behavioral  requirements  during  the  task  analysis  phase  of 
course  design.  It  was  important  to  develop  techniques  that  could  easily  be  incorporated  into  existing 
analysis  procedures.  In  addition,  because  existing  Cognitive  Task  Analysis  techniques  tend  to 
require  extensive  training  and  considerable  time  and  resources  to  implement,  there  was  a  need  to 
develop  streamlined  techniques  that  could  be  learned  and  applied  quickly  and  easily  due  to  the  time 
pressure  under  which  most  Instructional  Designers  work. 

This  Klein  Associates  project  originated  as  Phase  I  Small  Business  Innovative  Research 
contract.  The  initial  6-month  effort  resulted  in  a  prototype  streamlined  Cognitive  Task  Analysis 
approach.  This  report  documents  the  ensuing  Phase  n,  31 -month  effort  to  refine  and  finalize  the 
streamlined  approach  to  Cognitive  Task  Analysis;  evaluate  the  approach  to  assess  its  usability, 
validity,  and  reliability;  and  develop  stand-alone  instruction  so  that  the  approach  could  be  widely 
disseminated.  In  the  process  of  accomplishing  these  objectives,  an  analysis  of  the  potential  user 
population  was  conducted  to  ensure  that  the  streamlined  approach  to  Cognitive  Task  Analysis 
would  in  fact  fulfill  a  need  in  the  user  community  and  be  presented  in  a  usable  format.  In  order  to 
assess  the  streamlined  Cognitive  Task  Analysis  tools,  a  6-hour  workshop  was  developed  so  that  the 
tools  could  be  presented  to  a  number  of  potential  user  populations  for  both  informal  feedback  and  a 
formal  evaluation  study.  The  four  major  accomplishments  of  this  effort  are  as  follows: 

■  •  The  development  of  a  set  of  Applied  Cognitive  Task  Analysis  (ACTA)  tools  that  elicit 
important  cognitive  aspects  of  expert  performance,  and  that  are  easier  to  learn  than  earlier 
Cognitive  Task  Analysis  methods. 


1 


•  The  development  of  a  stand-alone  instructional  package,  including  a  CD-based  multimedia 
instructional  tool,  a  pamphlet  outlining  the  content  of  the  CD,  and  job  aids  to  assist  in 
conducting  the  ACTA  techniques. 

•  The  execution  of  an  evaluation  study  providing  data  that  indicate  that  graduate  students 
using  the  ACTA  techniques  are  consistently  able  to  elicit  experience-based,  cognitive 
information  and  generate  accurate,  important  training  materials  based  upon  the  information 
obtained  using  ACTA. 

•  The  execution  of  a  software  review,  which  indicates  that  both  Navy  Instructional  System 
Specialists  (ISSs)  and  private  sector  Instructional  Designers  found  the  stand-alone 
instructional  package  to  be  easy  to  use  and  informative. 

For  the  purpose  of  introduction,  we  include  a  brief  description  of  each  of  the  ACTA  tools,  and 
an  analysis  tool  termed  the  Cognitive  Demands  Table  in  this  first  section.  Details  of  how  the  tools 
were  developed  and  evaluated,  as  well  as  a  description  of  workshops  and  multimedia  interactive 
software  designed  to  train  Instructional  Designers  to  use  the  ACTA  techniques  are  included  in  the 
body  of  the  report. 

Task  Diagram 

The  Task  Diagram  elicits  a  broad  overview  of  the  task  and  identifies  the  difficult  cognitive 
elements.  Although  this  preliminary  interview  offers  only  a  surface-level  view  of  the  cognitive 
elements  of  the  task,  it  enables  the  interviewer  to  focus  the  more  in-depth  interviews  (i.e.,  the 
Knowledge  Audit  and  Simulation  Interviews)  so  that  time  and  resources  can  be  spent  unpacking  or 
uncovering  the  most  difficult  and  relevant  of  those  cognitive  elements. 

The  Subject  Matter  Expert  (SME)  is  asked  to  decompose  the  task  into  steps  or  subtasks.  The 
interviewer  limits  the  SME  to  between  three  and  six  steps,  to  ensure  that  time  is  not  wasted  delving 
into  minute  detail  during  the  surface-level  interview.  After  the  steps  of  the  task  have  been 
articulated,  the  SME  is  asked  to  identify  which  of  the  steps  require  cognitive  skill.  The  resulting 
diagram  serves  as  a  roadmap  for  future  interviews,  providing  an  overview  of  the  major  steps 
involved  in  the  task  and  the  sequence  in  which  the  steps  are  carried  out,  as  well  as  which  of  the 
steps  require  the  most  cognitive  skill. 

The  Task  Diagram  interview  is  intended  to  provide  a  surface-level  look  at  the  task,  and  does  not 
attempt  to  unpack  or  reveal  the  mental  model  of  each  SME.  Efforts  to  delineate  a  mental  model  can 
quickly  degenerate  into  a  search  for  everything  in  a  person’s  head  as  Rouse  and  Morris  (1986)  have 
pointed  out.  In  this  interview,  we  recommend  that  the  SME  be  limited  to  six  steps  and  advise  the 
interviewer  not  to  get  dragged  down  to  a  level  of  detail  that  is  best  captured  by  other  interview 
techniques. 
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Knowledge  Audit 

The  Knowledge  Audit  identifies  ways  expertise  is  used  in  a  domain  and  provides  examples 
based  on  actual  experience.  The  Knowledge  Audit  draws  directly  from  the  research  literature  on 
expert-novice  differences  and  expert  decision  making.  The  Knowledge  Audit  has  been  developed  as 
a  means  for  capturing  the  most  important  aspects  of  expertise  while  streamlining  the  intensive  data 
collection  and  analysis  methods  that  typify  studies  of  expertise. 

The  Knowledge  Audit  is  organized  around  knowledge  categories  that  have  been  found  to 
characterize  expertise.  These  include:  diagnosing  and  predicting,  situation  awareness,  perceptual 
skills,  developing  and  knowing  when  to  apply  tricks  of  the  trade,  improvising,  metacognition, 
recognizing  anomalies,  and  compensating  for  equipment  limitations.  Clearly,  we  could  have 
included  many  more  items,  but  our  intent  was  to  aim  for  the  smallest  number  of  high  impact 
components. 

The  Knowledge  Audit  employs  a  set  of  probes  designed  to  describe  types  of  domain  knowledge 
or  skill  and  elicit  appropriate  examples.  The  goal  is  not  simply  to  find  out  whether  each  component 
is  present  in  the  task,  but  to  find  out  the  nature  of  these  skills,  specific  events  where  they  were 
required,  strategies  that  have  been  used,  and  so  forth.  The  list  of  probes  is  the  starting  point  for 
conducting  this  interview.  Then,  the  interviewer  asks  for  specifics  about  the  example  in  terms  of 
critical  cues  and  strategies  of  decision  making.  This  is  followed  by  a  discussion  of  potential  errors 
that  a  novice,  less-experienced  person  might  have  made  in  this  situation. 

The  examples  elicited  with  the  Knowledge  Audit  do  not  contain  the  extensive  detail  and  sense 
of  dynamics  that  more  labor  intensive  methods  such  as  the  Critical  Decision  method  (Klein, 
Calderwood,  &  MacGregor,  1989)  incident  accounts  often  do.  However,  they  do  provide  enough 
detail  to  retain  the  appropriate  context  of  the  incident.  It  is  not  expected  that  all  probes  will  be 
equally  relevant  in  each  domain.  After  a  few  interviews,  interviewers  can  easily  determine  which 
probes  have  the  highest  payoff.  Although  the  Knowledge  Audit  does  not  capture  the  depth  of 
relationship  of  a  Conceptual  Graph  Structure  or  other  intensive  methods,  it  does  address  a  full  range 
of  aspects  of  expertise  that  are  usually  neglected  by  behavioral  task  analytic  methods. 

The  output  of  the  Knowledge  Audit  is  a  table  containing  an  inventory  of  task-specific  expertise. 
This  table  includes  examples  of  situations  in  which  experience  has  been  called  into  play,  cues  and 
strategies  used  in  dealing  with  these  difficult  situations,  and  an  explanation  why  such  situations 
present  a  challenge  to  less-experienced  operators. 

Simulation  Interview 

The  Simulation  Interview  allows  the  interviewer  to  better  understand  the  SME’s  cognitive 
processes  within  the  context  of  an  incident.  In  operational  settings,  the  point  of  the  job  is  typically 
to  act  upon  the  environment  in  some  manner.  Behavioral  task  analyses  are  designed  to  specify  the 
type  and  sequence  of  actions  involved  in  performing  a  task,  but  say  little  about  the  judgment  and 
decision-making  requirements,  such  as  troubleshooting,  diagnosis,  situation  assessment,  sensitivity 
to  critical  cues,  and  selection  of  courses  of  action  within  that  environment.  The  research  literature 
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has  shown  judgment  and  decision-making  processes  to  be  central  to  proficient  performance  of 
complex  tasks  (Howell  &  Cooke,  1989;  Klein,  1993).  Identification  and  exploration  of  information 
surrounding  high  consequence,  difficult  decisions  can  provide  a  sound  basis  for  generation  of 
effective  training  and  system  design. 

The  Simulation  Interview  is  based  on  presentation  of  a  challenging  scenario  to  the  SME.  We 
recommend  the  interviewer  retrieve  a  scenario  that  already  exists  for  use  in  this  interview.  Often, 
simulations  and  scenarios  exist  for  training  purposes.  It  may  be  necessary  to  adapt  or  modify  the 
scenario  to  conform  to  practical  constraints  such  as  time  limitations.  Developing  a  new  simulation 
specifically  for  use  in  the  interview  is  not  a  trivial  task  and  is  likely  to  require  an  up-front  Cognitive 
Task  Analysis  in  order  to  gather  the  foundational  information  needed  to  present  a  challenging 
situation.  The  simulation  can  be  in  the  form  of  a  paper-and-pencil  exercise,  perhaps  using  maps  or 
other  diagrams.  In  some  settings,  it  may  be  possible  to  use  video  or  computer-supported 
simulations.  Surprisingly,  in  our  experience,  fidelity  is  not  an  important  issue.  The  key  is  that  the 
simulation  present  a  challenging  scenario. 

After  exposure  to  the  simulation,  the  SME  is  asked  to  identify  major  events,  including 
judgments  and  decisions.  Each  event  is  probed  for  situation  assessment,  actions,  critical  cues,  and 
potential  errors  surrounding  that  event.  Using  the  same  simulation  for  interviews  with  multiple 
SMEs  can  provide  insight  into  situations  in  which  more  than  one  action  would  be  acceptable,  and 
alternative  assessments  of  the  same  situation  are  plausible.  This  technique  can  be  used  to  highlight 
differing  SME  perspectives,  which  is  important  information  for  developing  training  and  system 
design  recommendations.  The  technique  can  also  be  used  to  contrast  expert  and  novice  perspectives 
by  conducting  interviews  with  people  of  differing  levels  of  expertise  using  the  same  simulation. 

Cognitive  Demands  Table 

After  conducting  ACTA  interviews  with  multiple  SMEs,  we  recommend  the  use  of  a  Cognitive 
Demands  Table  to  sort  through  and  analyze  the  data.  Clearly,  not  every  bit  of  information  discussed 
in  an  interview  will  be  relevant  for  the  goals  of  a  specific  project.  The  Cognitive  Demands  Table  is 
intended  to  provide  a  format  for  the  practitioner  to  use  in  focusing  the  analysis  on  project  goals.  We 
offer  sample  headings  for  the  table  based  on  analyses  we  have  conducted  in  the  past  (difficult 
cognitive  elements,  why  difficult,  common  errors,  and  cues  and  strategies  used),  but  recommend 
that  practitioners  focus  on  the  types  of  information  they  will  need  to  develop  a  new  course  or  design 
a  new  system.  The  table  also  helps  the  practitioner  see  common  themes  in  the  data,  as  well  as 
conflicting  information  given  by  multiple  SMEs. 

The  report  is  organized  as  follows:  The  User  Analysis  section  describes  our  early  analysis  of  the 
potential  user  population.  The  Evolution  of  ACTA  section  details  the  evolution  of  the  ACTA  tools 
and  the  factors  that  impacted  the  development  of  the  tools.  The  ACTA  Workshops  section  includes  a 
description  of  the  workshop  used  to  teach  graduate  students  to  use  the  ACTA  techniques  so  that  we 
could  evaluate  the  methods,  and  a  discussion  of  how  the  ACTA  workshops  were  adapted  for 
different  audiences.  In  the  Evaluation  Study  section,  a  detailed  report  of  the  evaluation  study 
conducted  to  assess  the  usability,  validity,  and  reliability  of  the  ACTA  methods  is  provided.  The 
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final  portion  of  the  project  required  the  development  of  stand-alone  instructional  materials  for  the 
ACTA  techniques.  A  full  description  of  these  materials  and  the  rationale  for  the  design  of  the 
materials  is  included  in  the  Stand-Alone  Instruction  section.  The  final  section.  Conclusions  and 
Future  Research,  contains  conclusions  from  this  project  and  recommendations  for  future  research. 
In  addition,  five  appendices  are  included  containing  a  review  of  the  relevant  expertise  literature, 
sample  materials  from  our  workshops,  additional  data  from  the  evaluation  study,  ACTA  Software 
Pamphlet  and  the  ACTA  Job  Aids,  and  Instructions  from  the  Software  Review. 

User  Analysis 

The  development  of  the  ACTA  tools  began  in  Klein  Associates’  Phase  I  effort  of  this  study  and 
was  finalized  in  the  Phase  II  effort  documented  in  this  report.  In  the  Phase  I  effort,  candidate 
Cognitive  Task  Analysis  tools  were  identified  based  upon  a  survey  of  existing  Cognitive  Task 
Analysis  techniques  and  our  own  experience  conducting  Cognitive  Task  Analysis.  The  Phase  13 
effort  began  with  an  analysis  of  our  intended  user  population.  Although  U.S.  Navy  ISSs  were 
identified  as  our  primary  user  population,  the  needs  of  private  sector  Instructional  Designers  were 
also  investigated.  A  secondary  goal  of  this  project  was  to  build  streamlined  Cognitive  Task 
Analysis  techniques  that  could  be  commercialized  beyond  the  Navy. 

The  reason  for  conducting  the  user  analysis  was  to  ensure  that  the  ACTA  techniques  would  be 
useful  to  our  target  audience.  We  interviewed  four  Navy  ISSs,  three  Navy  instructors,  and  three 
private  sector  Instructional  Designers.  In  these  interviews,  we  explored  each  interviewee’s  current 
job  responsibilities,  the  types  of  courses  developed  and  revised,  and  the  tools  currently  used  to 
capture  the  task  for  which  a  course  was  being  developed.  In  addition,  we  presented  our  ideas  for  the 
ACTA  techniques.  At  the  time  of  the  user  analysis,  the  ACTA  techniques  included  the  Knowledge 
Audit  and  Simulation  Interview,  which  are  described  in  the  introduction  to  this  paper,  and  a  Team 
Schematic  Interview  aimed  at  unpacking  team  interactions  and  information  flow.1  Our  intent  was  to 
better  understand  the  environment  in  which  our  potential  user  population  worked,  find  out  where 
existing  tools  were  lacking,  and  gauge  the  acceptability  of  the  candidate  Cognitive  Task  Analysis 
techniques  identified  in  the  Phase  I  effort. 

This  analysis  revealed  that  both  ISSs  and  private  sector  Instructional  Designers  work  under  time 
pressure  in  developing  courses.  Many  indicated  that  they  would  like  access  to  tools  that  would 
allow  them  to  capture  the  cognitive  elements  of  a  task,  but  that  these  tools  would  have  to  be  highly 
efficient  if  they  were  to  be  used  at  all.  Details  about  the  environments  in  which  our  interviewees 
work  and  reactions  to  our  candidate  Cognitive  Task  Analysis  techniques  are  included  below. 


‘The  Team  Schematic  Interview  was  abandoned  as  one  of  the  ACTA  techniques  because  feedback  from  potential 
users  indicated  that  it  would  not  be  generally  applicable.  However,  the  need  expressed  for  such  a  tool  in  domains  where 
team  issues  are  key  indicates  that  additional  work  in  the  area  of  team  Cognitive  Task  Analysis  tools  would  be  fruitful. 
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Navy  ISSs  work  in  conjunction  with  instructors  to  develop  and  revise  courses.  The  instructors 
are  Subject  Matter  Experts  (SMEs)  for  the  course  they  will  teach.  For  example,  an  experienced 
Electronic  Warfare  (EW)  technician  will  serve  as  an  instructor  for  the  EW  school.  ISSs  thus  rely  on 
the  instructors  for  content  knowledge  and  expertise  in  performing  the  task  to  be  taught.  ISSs  work 
with  instructors  to  capture  this  knowledge  and  formalize  it  into  course  materials. 

Few  new  courses  are  developed  in  the  Navy.  Completely  new  jobs  rarely  arise  in  the  Navy.  It  is 
much  more  common  for  an  existing  job  to  evolve,  adding  or  replacing  skills  as  new  equipment  and 
new  tactics  become  available.  Thus,  ISSs  and  instructors  spend  the  bulk  of  their  time  revising 
existing  courses  to  adapt  to  the  changing  state  of  the  world.  In  situations  in  which  a  completely  new 
piece  of  equipment  is  developed,  the  contractor  who  built  the  equipment  generally  delivers  a  first- 
generation  course  on  how  to  operate  the  equipment.  Future  generations  of  the  course  are  revised  and 
adapted  by  the  Navy  ISSs  and  instructors. 

The  most  recent  publications  describing  the  tools  currently  used  by  the  Navy  to  revise  courses 
and  develop  course  materials  are  the  NAVEDTRA  130/131  series.  These  documents  clearly  lay  out 
how  to  apply  the  Instructional  System  Design  process  (Gagne  &  Briggs,  1974)  within  the  context  of 
the  Navy.  These  materials  do  include  information  about  how  to  gather  content  information  for  a 
course.  Suggested  methods  include  examining  existing  materials  relating  to  the  task,  on-site 
observations/interviews,  and  a  jury  of  experts.  Although  interviewing  experts  is  suggested,  little 
guidance  is  offered  as  to  how  to  conduct  the  interviews  or  what  types  of  information  to  explore.  In 
addition,  a  somewhat  discouraging  note  is  included  “...interview  methods  require  considerable 
effort.  Because  of  the  massive  amounts  of  data  and  information  that  must  be  tabulated  and 
summarized,  you  may  not  have  the  resources  to  implement  these  methods,  (pp.  3-10).”  The  result  is 
that  ISSs  tend  to  rely  on  existing  materials  describing  the  job  and  the  ability  of  the  instructor  (who 
is  also  an  SME)  to  unpack  his/her  own  expertise. 

There  is  a  wide  range  in  terms  of  cognitive  content  for  the  courses  the  Navy  offers.  It  is  not 
necessarily  true  that  every  Navy  course  would  warrant  a  Cognitive  Task  Analysis.  The  ISSs  we 
interviewed  indicated  that  they  could  easily  judge  which  of  the  courses  they  oversee  require  difficult 
cognitive  skills  and  are  critical  enough  to  make  Cognitive  Task  Analysis  a  worthwhile  endeavor. 

ISSs  see  a  need  to  get  more  cognitive  information  into  the  courses  that  are  offered  by  the  Navy. 
The  current  process  encourages  instructors  to  “personalize”  a  course,  which  consists  of  going 
through  and  adding  illustrations  and  exercises  that  will  highlight  difficult  cognitive  elements  of  the 
job  based  on  that  instructor’s  experience  on  the  job.  Although  this  personalization  process  tends  to 
work  well,  it  is  idiosyncratic  in  that  the  sea  stories  and  exercises  offered  vary  from  one  instmctor  to 
the  next,  depending  on  his/her  personal  experience.  As  a  result,  some  instructors  may  make  good 
use  of  sea  stories  to  provide  context  and  illustrate  difficult  judgments,  decisions,  perceptual  skills, 
metacognition,  etc.  In  other  cases,  sea  stories  may  simply  serve  as  entertainment  with  little 
instructional  value.  ISSs  recognize  that  course  tools  that  would  allow  cognitive  skills  to  be 
addressed  consistently,  regardless  of  the  instructor,  are  needed. 


In  general,  the  Navy  instructors  were  more  critical  of  the  concept  of  capturing  expertise  and 
including  it  in  courses  than  the  ISSs.  The  instructors  believe  that  cognitive  skills  and  expertise  are 
learned  on  the  ship  via  on-the-job  training.  As  products  of  the  current  Navy  training  system,  they 
believe  that  it  works  well.  There  is  a  fear  that  to  try  to  document  expertise  and  cognitive  skill,  so 
that  it  can  be  included  in  a  course,  will  result  in  a  list  of  procedures  that  do  not  adequately  capture 
the  concepts  and  skills.  The  result  would  be  additional  coursework  with  little  relevance  to  the  actual 
job. 


Private  sector  instructors  voiced  a  frustration  that  cognitive  skills  are  not  captured  well  and  are 
hard  to  teach.  The  need  for  tools  such  as  ACTA  was  recognized  by  most  of  our  interviewees.  Some, 
however,  echoed  the  concerns  of  the  Navy  instructors  that  these  types  of  skills  can  only  be  acquired 
via  experience. 

Financial  concerns  also  play  a  role  in  the  Navy’s  training  environment.  There  is  currently  a  push 
to  reduce  time  spent  in  training.  Training  budgets  are  currently  very  tight.  The  need  to  offer  quality 
courses  in  as  little  time  as  possible  is  critical.  It  would  be  important  that  the  ACTA  tools  not  result 
in  increased  training  time. 

In  addition  to  pressure  to  use  course  time  efficiently,  private  sector  Instructional  Designers  have 
considerable  pressure  to  reduce  the  time  needed  to  develop  a  course.  The  fact  that  time  pressure  is 
an  important  constraint  for  both  Navy  and  private  sector  Instructional  Designers  proved  to  be  a 
guiding  principle  for  us  as  we  continued  to  refine  the  ACTA  techniques.  The  need  to  balance  the 
thoroughness  and  depth  of  the  methods  we  developed  with  the  time  constraints  of  our  intended 
users  was  a  constant  tension  throughout  the  method  development. 

Both  ISSs  and  instructors  resonated  to  the  idea  of  an  interview  technique  termed  the  Simulation 
Interview  that  would  be  structured  around  a  simulation.  Scenarios  and  simulations  are  currently 
used  in  many  Navy  courses.  The  use  of  our  cognitive  probes  to  help  SMEs  unpack  their  own 
processes  as  they  approach  a  task  was  met  with  enthusiasm.  In  addition,  instructors  saw  these 
probes  as  a  potential  teaching  tool. 

In  our  interviews  with  Navy  personnel,  the  idea  of  an  interview  technique  aimed  at  unpacking 
team  interactions  and  information  flow  got  mixed  reviews,  depending  on  the  type  of  course  for 
which  the  interviewee  was  responsible.  For  jobs  that  require  information  flow  among  several  team 
members,  a  team-focused  technique  was  viewed  as  a  potentially  useful  tool  for  capturing  and 
representing  information.  However,  many  of  the  people  we  interviewed  were  not  working  on 
courses  for  which  information  flow  was  a  critical  element. 

The  description  of  the  Knowledge  Audit,  a  technique  that  probes  specific  aspects  of  expertise, 
evoked  a  positive  reaction  from  the  Navy  ISSs  and  instructors.  In  these  interviews,  discussions  of 
specific  probes  helped  us  to  cull  the  long  list  of  probes  we  started  with  for  the  Knowledge  Audit  to  a 
set  that  could  be  easily  learned  and  implemented  in  a  1-hour  interview. 

Reactions  to  the  ACTA  techniques  were  generally  favorable  in  the  private  sector.  Many  of  the 
Instructional  Designers  we  spoke  with  had  developed  their  own  methods  with  many  of  the  elements 
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we  suggested  via  experience.  They  confirmed  that  to  document  and  disseminate  these  types  of 
techniques  would  be  of  value  to  many  Instructional  Designers  as  they  search  for  methods  that  will 
help  to  capture  cognitive  skills. 


Evolution  of  ACTA 

This  section  describes  the  rationale  behind  each  of  the  ACTA  tools  and  the  Cognitive  Demands 
Table.  In  addition,  the  evolution  of  each  technique  is  described. 

The  methods  presented  here  are  complementary;  each  is  designed  to  get  at  different  aspects  of 
cognitive  skill.  The  first  technique,  the  Task  Diagram  interview,  provides  the  interviewer  with  a 
broad  overview  of  the  task  and  highlights  the  difficult  cognitive  portions  of  the  task  to  be  probed 
further  with  in-depth  interviews.  The  second  technique,  the  Knowledge  Audit,  surveys  the  aspects 
of  expertise  required  for  a  specific  task  or  subtask.  As  each  aspect  of  expertise  is  uncovered,  it  is 
probed  for  concrete  examples  in  the  context  of  the  job,  cues  and  strategies  used,  and  why  it  presents 
a  challenge  to  inexperienced  people.  The  third  technique,  the  Simulation  Interview,  allows  the 
interviewer  to  probe  the  SMEs  cognitive  processes  within  the  context  of  a  specific  scenario.  The  use 
of  a  simulation  or  scenario  provides  job  context  that  is  difficult  to  obtain  via  the  other  interview 
techniques,  and  therefore  allows  additional  probing  around  issues  such  as  situation  assessment,  how 
situation  assessment  impacts  a  course  of  action,  and  potential  errors  a  novice  would  be  likely  to 
make  given  the  same  situation.  Finally,  a  Cognitive  Demands  Table  is  offered  as  a  means  to 
consolidate  and  synthesize  the  data,  so  that  it  can  be  directly  applied  to  a  specific  project. 

Building  a  Road  Map:  The  Task  Diagram 

The  first  component  of  ACTA  is  the  Task  Diagram  (Figure  1).  Its  function  is  to  embed  the 
Cognitive  Task  Analysis  within  the  task  and  to  focus  the  rest  of  the  interviews  on  the  cognitively 
challenging  parts  of  the  task.  The  expectation  is  that  one  would  undertake  a  Cognitive  Task 
Analysis  in  the  context  of  a  specific  project  for  which  understanding  the  cognitive  components  of  a 
task  is  necessary.  The  goals  of  the  project  would  thus  determine  which  task  (or  set  of  tasks)  should 
be  studied.  Cognitive  Task  Analysis  is  most  valuable  for  tasks  that  have  an  important  cognitive 
component.  To  some  extent,  the  Task  Diagram  is  a  mental  model  of  how  the  SME  views  the  task, 
but  it  is  very  simplified.  We  recommend  limiting  it  to  six  elements  in  order  to  obtain  a  broad 
overview  of  the  task.  This  interview  does  not  provide  the  structure  and  probes  needed  to  obtain  an 
in-depth  detailed  account  of  the  task.  Initially,  the  Task  Diagram  included  two  stages  of 
decomposition.  Each  cognitive  element  identified  in  the  initial  Task  Diagram  became  the  topic  for 
its  own  Task  Diagram.  We  found  that  this  added  a  layer  of  complexity  with  little  payoff. 
Participants  in  our  workshops  found  that  the  initial  Task  Diagram  was  sufficient  to  provide  a  broad 
overview  of  the  task  and  identify  the  cognitive  elements  to  be  probed  in  later  interviews.  We  do 
allow  for  the  likelihood  that  incidents  will  arise  in  which  the  initial  Task  Diagram  will  not  provide 
sufficient  information  for  the  interviewer  to  progress  with  the  Cognitive  Task  Analysis.  In  such 
cases,  we  would  recommend  a  second  stage  of  decomposition.  This  decision  is  left  to  the 
interviewer’s  discretion. 
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We  considered  other  approaches  here,  specifically  concept  mapping  and  team  schematics,  but 
these  proved  impractical.  We  had  used  concept  maps  in  a  number  of  projects.  Although  we  found 
the  information  gathered  via  concept  mapping  to  be  useful,  the  technique  was  difficult  for  some 
researchers  to  learn  and  especially  difficult  to  standardize  across  researchers.  The  resulting  concept 
maps  were  often  idiosyncratic  and  difficult  to  communicate  to  people  who  did  not  participate  in  the 
interview. 


Getting  the  call 


* 


Figure  1.  Task  diagram  of  Fire  Ground  Commander’s  job  in  commanding  crew. 

We  also  considered  a  team  schematic  that  would  capture  the  major  elements  of  the  task,  identify 
the  team  members,  and  outline  the  flow  of  information  among  the  team  members  for  each  major 
task  element.  This  technique  was  abandoned  early  in  the  project  as  our  user  analysis  indicated  that 
team  elements  and  information  flow  were  key  issues  for  only  a  subset  of  domains  for  which  Navy 
ISSs  are  responsible. 

Capturing  Aspects  of  Expertise:  The  Knowledge  Audit 

The  research  on  the  nature  of  expertise  and  expert-novice  differences  indicates  that  skilled 
performance  involves  a  number  of  factors  over  and  above  the  facts  and  rules  a  person  has  learned. 
These  factors  must  be  taken  into  account  if  training  systems  are  intended  to  develop  or  enhance 
cognitive  skills  necessary  to  attain  expertise. 

The  Knowledge  Audit  was  developed  to  focus  on  the  following  factors  that  contribute  to 
expertise — to  elicit  information  about  diagnosing  and  predicting,  situation  awareness,  perceptual 
skills,  developing  and  knowing  when  to  apply  tricks  of  the  trade,  improvising,  metacognition, 
recognizing  anomalies,  and  compensation  for  equipment  limitations.  The  Knowledge  Audit  is  a 
straightforward  technique  that  asks  the  SME  how  each  of  these  aspects  is  involved  in  the  task  being 
studied  and  poses  follow-up  questions  regarding  the  cues  and  strategies  an  expert  relies  on  and  what 
presents  difficulty  for  inexperienced  people  (Figure  2).  The  Knowledge  Audit  identifies  ways 
expertise  is  used  in  a  domain  and  provides  examples  based  on  actual  experience. 

The  Knowledge  Audit  draws  directly  from  the  research  literature  on  expert-novice  differences 
and  our  own  Critical  Decision  method  studies  of  expert  decision  making.  (See  Appendix  A  for  a 
review  of  the  expertise  literature,  which  impacted  the  development  of  the  Knowledge  Audit.)  We 
have  developed  the  Knowledge  Audit  as  a  means  for  capturing  the  most  important  aspects  of 
expertise  while  streamlining  the  intensive  data  collection  and  analysis  methods  that  typify  studies  of 
expertise.  Specifically,  the  Knowledge  Audit  probes  were  based  on  an  understanding  of  how  experts 
make  decisions.  There  are  several  aspects  to  expertise  in  decision  making.  The  Knowledge  Audit 
probes  are  based  around  these  aspects  of  expertise. 
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•  Past  and  Future  Probe  (Diagnosing  and  Predicting,  Mental  Simulation).  When 
experiencing  a  situation,  experts  are  able  to  figure  out  how  that  situation  developed,  and 
they  can  think  into  the  future  to  see  where  the  situation  is  going.  Among  other  things,  this 
allows  the  expert  to  head  off  problems  before  they  develop  (de  Groot  1946/1978;  Endsley, 
1995;  Klein  &  Crandall,  1995;  Klein  &  Hoffman,  1993). 

•  Big  Picture  Probe  (Situation  Awareness).  Novices  may  only  see  bits  and  pieces  of  a 
situation.  Experts  are  able  to  quickly  build  an  understanding  of  the  whole  situation,  the  “big 
picture”  view.  This  allows  the  expert  to  think  about  how  different  elements  fit  together  and 
affect  each  other  (Endsley,  1995;  Klein,  1997). 


Aspects  of  Expertise 

Cues  and  Strategies 

Why  Difficult? 

Past  and  Future 

E.g.,  explosion  in  office  strip: 
search  the  office  areas  rather  than 
source  of  explosion. 

Material  safety  data  sheets  (MSDS) 
tell  you  that  explosion  in  area  of 
dangerous  chemicals  and 

information  about  chemicals. 

Start  where  most  likely  to  find 
victims  and  own  safety 

considerations. 

Novice  would  be  trained  to  start  at 
source  and  work  out. 

May  not  look  at  MSDS,  to  find 
potential  source  of  explosion,  and 
account  for  where  people  are  most 
likely  to  be. 

Big  Picture 

Big  picture  includes  source  of 
hazard,  potential  location  of  victims, 
ingress/egress  routes,  other  hazards. 

Senses,  communication  with  others, 
building  owners,  MSDS,  building 
pre-plans. 

Novice  gets  tunnel  vision,  focuses 
on  one  thing  (e.g.,  victims). 

Noticing 

Breathing  sounds  of  victims. 

Both  you  and  partners  stop,  hold 
your  breath,  and  listen. 

Listen  for  crying,  talking  to 
themselves,  victims  knocking  things 
over. 

Noise  from  own  breathing  in 
apparatus,  fire  noises. 

Don’t  know  what  kinds  of  sounds  to 
listen  for. 

Operating  without  vision. 

Keep  in  touch  with  wall,  feel  for 
windows/doors,  means  of  egress. 

Use  touch  more,  feeling  for  objects 
at  head  level,  on  ground,  rope  to 
solid  objects  outside  building. 

Garage  provides  more  hazards; 
knowing  hazards  in  building  types. 

Listen  more  actively. 

Navigation  through  building, 

avoiding  hazards,  get  used  to 
working  for  30-60  minutes  without 
light. 

Figure  2.  Knowledge  Audit  of  search  and  rescue  procedure  with  Fireground  Commander. 

•  Noticing  Probe  (Perceptual  Skills).  Experts  are  able  to  detect  cues  and  see  meaningful 
patterns  that  less-experienced  personnel  may  miss  altogether  (de  Groot,  1946/1978;  Klein  & 
Hoffman,  1993;  Shanteau,  1985). 
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•  Job  Smarts  Probe  (Heuristics,  Tricks  of  the  Trade,  Contextual  Practices).  Experts  learn 
how  to  combine  procedures  and  work  the  task  in  the  most  efficient  way  possible.  They  do 
not  cut  comers,  but  they  do  not  waste  time  and  resources  either  (Klein  &  Hoffman,  1993). 

•  Opportunities/Improvising  Probe.  Experts  are  comfortable  improvising — seeing  what  will 
work  in  this  particular  situation.  They  are  also  able  to  shift  directions  to  take  advantage  of 
opportunities  (Dreyfus  &  Dreyfus,  1986;  Shanteau,  1985). 

•  Self-monitoring  Probe  (Metacognition).  Experts  are  aware  of  their  performance,  they 
check  how  they  are  doing  and  make  adjustments.  Experts  notice  when  their  performance  is 
not  what  it  should  be  (due  to  stress  or  fatigue,  for  example),  and  they  are  able  to  adjust  to  get 
the  job  done  (Cohen,  Freeman,  &  Wolf,  1996;  Glaser  &  Chi,  1988). 

•  Anomalies  Probe  (Perceptual  Skills,  Situation  Awareness).  Novices  don’t  know  what  is 
typical  so  they  have  a  hard  time  identifying  the  unusual  or  atypical.  Experts  can  quickly  spot 
unusual  events  and  detect  deviations,  and  they  are  able  to  notice  when  something  that  should 
happen  does  not  happen  (Chi,  Hutchinson,  &  Robin,  1988;  Klein,  1989;  Klein,  1997;  Klein 
&  Hoffman,  1993). 

•  Equipment  Difficulties  Probe.  Experts  know  that  their  equipment  can  sometimes  mislead. 
Novices  usually  believe  whatever  the  equipment  tells  them,  they  don’t  know  to  be  skeptical 
(Cannon-Bowers,  Salas,  &  Converse,  1992). 

Changes  made  to  the  Knowledge  Audit  method  throughout  this  project  consisted  primarily  of 
minor  refinements.  It  has  been  used  throughout  the  course  of  the  project  by  Klein  Associates’ 
interviewers  who  have  found  it  to  be  a  useful  tool.  It  was  also  used  by  graduate  students,  novice 
interviewers,  as  a  further  testbed.  There  were  two  primary  alterations  to  the  method.  The  first  was  to 
reduce  the  number  of  probes  so  that  the  Knowledge  Audit  would  accommodate  the  necessary  time 
constraints.  The  second  was  to  eliminate  jargon  and  incorporate  more  everyday  language  so  that  the 
interviewer  could  understand  the  concept  of  the  aspect  of  expertise  being  probed  and  also  so  that  the 
SMEs  could  understand  what  they  were  being  asked.  We  also  learned  that  providing  an  explanation 
of  each  probe  was  a  useful  way  to  introduce  the  question  and  to  elicit  more  concrete  responses 
based  on  the  SME’s  own  experiences.  These  explanations  were  added  to  the  probes  and  provided 
for  a  better  interview. 

Understanding  Decisions  and  Judgments:  The  Simulation  Interview 

The  Simulation  Interview  draws  on  the  experiences  of  the  SME  in  an  exercise  where  the  SME  is 
asked  to  use  his/her  expertise  to  provide  a  description  of  what  actions  s/he  would  take  at  each 
decision  point  in  a  specific  scenario,  followed  by  an  explanation  of  his/her  situation  assessment  at 
that  point  in  time,  what  information  s/he  is  relying  on  to  make  that  assessment,  and  errors  an 
inexperienced  person  might  make  in  that  situation  (Figure  3).  The  Simulation  Interview  draws  on 
the  Critical  Decision  method,  (Klein,  Calderwood,  &  MacGregor,  1989),  verbal  protocol  analysis, 
and  other  think-aloud  methods.  Simulation-and  incident-based  interviews  have  been  used 
successfully  in  many  domains  (Bell  &  Hardiman,  1989;  Clarke,  1987;  Cordingley,  1989;  Diederich, 
Ruhmann,  &  May,  1987;  Flanagan,  1954;  Grover,  1983;  Hall,  Gott,  &  Pokomy,  1995;  Klein, 
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Calderwood,  &  MacGregor,  1989;  Thordsen,  Militello,  &  Klein,  1992).  The  Simulation  Interview 
was  developed  based  on  our  experiences  using  simulations  and  scenarios  to  provide  the  background 
needed  to  understand  the  rich,  contextual  elements  of  a  task.  Critical  Decision  method  probes  have 
been  incorporated  into  this  technique. 


Events 

Actions 

Assessment 

Critical  Cues 

Potential  Errors 

On-scene  arrival 

Account  for  people 
(names). 

Ask  neighbors  (but 
don’t  take  their  word 
for  it,  check  it  out 
yourself). 

Must  knock  on  or 
knock  down  to  make 
sure  people  aren’t 
there. 

It’s  a  cold  night,  need 
to  find  place  for 
people  who  have  been 
evacuated. 

Night  time 

Cold  ->  15  degrees. 
Dead  space. 

Add  on  floor. 

Poor  materials:  wood 
(punk  board),  metal 
grinders  (buckle  an 
break  under  fire). 

Common  attic  in  whole 
building 

Not  keeping  track 
of  people  (could  be 
looking  for  people 
who  are  not  there). 

Initial  attack. 

Watch  for  signs  of 
building  collapse. 

If  signs  of  building 
collapse,  evacuate 

and  throw  from 
outside. 

Faulty  construction, 
building  may  collapse. 

Signs  of  building 
collapse  include:  What 
walls  are  doing 

(cracking). 

What  floors  are  doing 
(groaning). 

What  metal  girders  are 
doing  (clicking, 

popping). 

Cable  in  old  buildings 
hold  walls  together. 

Ventilating  the 

attic,  this  draws  the 
fire  up  and  spreads 
it  through  the  pipes 
and  electrical 

system. 

Figure  3.  Simulation  Interview  with  Fireground  Commander. 

One  of  our  concerns  for  this  part  of  the  ACTA  toolkit  was  that  the  Instructional  Designer  might 
have  trouble  generating  or  finding  a  scenario  or  simulation  capability.  However,  during  the 
development  of  ACTA  we  tried  several  different  types  of  scenarios  from  textual  case  studies,  which 
describe  a  scenario  and  various  developments  in  the  scenario,  to  slightly  more  complex  scenarios 
using  props  (such  as  maps  for  planning  courses  of  action  or  for  getting  an  idea  of  the  situation), 
through  high-fidelity  complex  simulations  that  are  sometimes  found  in  military  training  situations. 
Our  success  with  applying  scenarios  and  simulations  with  varying  degrees  of  fidelity  gave  us 
confidence  that  an  Instructional  Designer  would  be  able  to  find  or  generate  a  means  of  conducting 
this  interview. 

A  further  concern  was  in  the  nature  of  the  probes  to  use  at  each  decision  point  or  development  in 
the  scenario,  as  well  as  how  to  record  that  information.  We  worked  with  several  different  variations 
of  probes  as  well  as  a  number  of  different  table  types  for  recording  information.  The  final  probes 
and  table  formats  were  decided  upon  based  on  our  own  experiences  with  workshop  participants  and 
with  our  evaluation  study. 
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Knowledge  Representation:  Cognitive  Demands  Table 


Our  final  challenge  in  developing  the  ACTA  techniques  was  to  provide  a  method  that  would 
consolidate  the  data  gathered  via  the  three  ACTA  techniques  and  multiple  SMEs  to  a  representation 
that  would  capture  the  cognitive  aspects  of  the  task  as  well  as  providing  meaningful  input  into 
training  materials.  We  decided  to  use  another  table  format  to  capture  the  themes  and  high  level 
commonalities  across  interviews  (Figure  4).  This  representation  grew  from  our  own  efforts  to  come 
up  with  Decision  Requirements  Tables,  or  Decision  Requirements  Inventories.  These  tables  were 
intended  to  represent  the  critical  decisions  and  judgments  in  a  task  and  how  these  decisions  and 
judgments  are  made.  These  tables  provide  the  Instructional  Designer  with  the  means  to  structure 
and  analyze  the  data.  The  column  headings  are  not  rigid,  they  can  be  adapted  to  suit  the  goals  of  the 
project  and  to  capture  information  that  the  designer  finds  useful  for  training  recommendations  or 
interventions.  We  adapted  these  tables  in  order  to  allow  the  Instructional  Designer  to  pull 
information  from  multiple  interviews  into  a  single  consolidated  representation  that  could  serve  as  a 
permanent  record  of  the  Cognitive  Task  Analysis.  These  records  can  serve  as  a  reference  for  future 
training  interventions  for  cognitive  skills  training.  Other  means  of  recording  these  data  may  be  in 
the  form  of  the  examples  and  incidents  that  were  recorded  as  parts  of  the  Knowledge  Audit 
interview.  These  stories  could  be  used  as  a  means  to  capture  sea  and  war  stories  as  well  as  form  the 
basis  for  cognitively  challenging  training  exercises. 


Difficult  Cognitive 
Element 

Why  Difficult? 

Common  Errors 

Cues  and  Strategies 

Used 

Knowing  where  to 
search  after  an 

explosion 

Novices  may  not  be 
trained  in  dealing  with 
explosions.  Other 

training  suggests  you 
should  start  at  the  source 
and  work  outward. 

Not  everyone  knows 
about  the  Material  Safety 
Data  Sheets.  These 

contain  critical 

information. 

Novice  would  be  likely  to 
start  at  the  source  of  the 
explosion.  Starting  at  the 
source  is  a  rule  of  thumb 
for  most  other  kinds  of 
incidents. 

Start  where  you  are  most  likely 
to  find  victims,  keeping  in  mind 
safety  considerations. 

Refer  to  Material  Safety  Data 
Sheets  to  determine  where 
dangerous  chemicals  are  likely 
to  be. 

Consider  the  type  of  structure 
and  where  victims  are  likely  to 
be. 

Consider  the  likelihood  of 
further  explosions.  Keep  in 
mind  the  safety  of  your  crew. 

Finding  victims  in  a 
burning  building. 

There  are  lots  of 
distracting  noises.  If  you 
are  nervous  or  tired, 
your  own  breathing 
makes  it  hard  to  hear 
anything  else. 

Novices  sometimes  don’t 
recognize  their  own 
breathing  sounds;  they 
mistakenly  think  they 
hear  a  victim  breathing. 

Both  you  and  your  partner  stop, 
hold  your  breath,  and  listen. 

Listen  for  crying,  victims 
talking  to  themselves,  victims 
knocking  things  over,  etc. 

Figure  4.  Cognitive  Demands  Table  example  derived  from  Fireground 
Command  interviews. 
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ACTA  Workshops 

Klein  Associates  conducted  workshops  throughout  the  course  of  this  project  in  order  to  teach 
people  the  Applied  Cognitive  Task  Analysis  (ACTA)  techniques  and  to  obtain  feedback  that  was 
used  to  refine  the  techniques.  Many  different  forms  of  the  workshop  were  developed  based  on  who 
the  participants  were  and  how  much  time  was  available  for  the  workshop  presentation.  A  primary 
driver  for  developing  formal  workshops  to  teach  the  ACTA  techniques  was  our  need  to  evaluate  the 
techniques.  The  workshops  were  used  to  teach  graduate  students  to  conduct  interviews  and  create 
representations  using  the  ACTA  techniques,  analyze  the  information  gathered,  and  develop 
instructional  materials  for  a  potential  course.  This  allowed  us  to  evaluate  the  usability,  validity,  and 
reliability  of  the  ACTA  techniques. 

In  this  section  of  the  report  we  first  describe  in  detail  the  workshops  conducted  as  part  of  the 
Evaluation  Study.  Following  the  description  of  the  workshops  developed  for  the  Evaluation  Study, 
we  briefly  describe  other  workshops  conducted  as  part  of  this  project  and  also  discuss  what  types  of 
workshops  work  best  for  teaching  people  to  use  the  ACTA  techniques. 

Workshops  for  the  Evaluation  Study 

The  workshops  developed  for  the  Evaluation  Study  were  the  most  intensive  workshops 
developed  during  this  project  because  our  participants  were  graduate  students  who  had  little  or  no 
background  in  interviewing  techniques  or  the  preparation  of  training  materials.  We  developed  two 
different  workshops  for  these  students:  an  introductory  workshop,  which  lasted  approximately  2 
hours,  and  an  ACTA  workshop,  which  lasted  approximately  6  hours.  Each  of  these  workshops  is 
described  in  turn. 

Introductory  Workshop 

The  purpose  of  this  workshop  was  fourfold: 

•  To  provide  the  students  with  an  overview  of  the  Evaluation  Study  in  which  they  were 
participating. 

•  To  review  basic  interview  skills. 

•  To  introduce  the  students  to  Cognitive  Task  Analysis. 

•  To  instruct  the  students  in  how  to  develop  the  training  materials  that  would  serve  as  the 
primary  dependent  measures  for  the  Evaluation  Study. 

The  Evaluation  Study  overview  included  background  information  on  the  purpose  and  sponsor 
for  the  project,  a  description  of  the  requirements  for  participating  in  the  project  (e.g.,  each  student 
would  participate  in  interviews  with  two  Subject  Matter  Experts  [SMEs]),  the  timeline  for  the 
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Evaluation  Study,  and  completion  of  an  informed  consent  form.  In  addition,  students  were  given  a 
brief  overview  of  the  domain  and  the  specific  task  they  would  be  investigating. 

The  review  of  basic  interviewing  skills  covered  topics  ranging  from  appropriate  dress  to 
fundamental  good  manners  (e.g.,  be  courteous  at  all  times,  thank  the  SME  for  his/her  help,  etc.).  We 
discussed  the  need  to  establish  rapport  at  the  beginning  of  the  interview  before  starting  with  their 
prepared  questions.  We  conducted  a  role  play  exercise  in  which  each  student  had  the  opportunity  to 
practice  introducing  him/herself  to  an  “SME.”  We  also  presented  some  background  information  on 
the  domain  in  which  the  SMEs  worked  (i.e.,  firefighters  or  EW  operators)  to  help  acquaint  the 
students  with  these  domains  prior  to  their  first  interviews. 

The  introduction  to  Cognitive  Task  Analysis  started  with  an  overview  of  the  field  of  Cognitive 
Science.  We  emphasized  that  cognitive  scientists  are  interested  in  the  thinking  underlying 
observable  behaviors.  We  contrasted  traditional  laboratory  methods  that  have  been  used  to  study 
cognitive  processes  with  more  applied  methods  such  as  field  observations  and  interviews.  We  also 
contrasted  traditional  approaches  to  studying  job  performance,  which  focuses  on  behavioral  aspects 
with  Cognitive  Task  Analysis  that  attempts  to  identify  the  critical  cognitive  factors  involved  in  a 
job. 


The  introductory  workshop  concluded  with  a  detailed  explanation  of  the  materials  the  students 
needed  to  produce  after  conducting  their  interviews:  a  Cognitive  Demands  Table,  learning 
objectives,  and  modifications  to  a  course  manual.  After  describing  the  types  of  information  to  be 
included  in  the  different  columns  of  the  Cognitive  Demands  Table,  a  concrete  example  based  on  an 
interview  with  a  car  mechanic  was  reviewed  (shown  in  Figure  5).  Because  the  student  participants 
had  no  prior  experience  developing  learning  objectives,  we  practiced  this  skill  in  the  introductory 
workshop.  Each  student  was  given  a  copy  of  Figure  6  that  lists  potential  action  verbs  to  be  included 
in  learning  objectives  and  was  asked  to  develop  learning  objectives  based  on  the  information  in  the 
Cognitive  Demands  Table  shown  in  Figure  5.  The  learning  objectives  written  by  the  students  were 
critiqued  by  the  group  in  order  to  help  the  students  learn  what  constitutes  a  good  learning  objective. 
In  order  to  prepare  the  students  to  develop  modifications  to  course  manuals,  an  example  involving 
modifications  to  an  auto  repair  manual  was  reviewed. 

Students  investigating  the  size-up  task  of  the  Fireground  Commander  were  given  written 
materials  describing  the  size-up  task  in  the  context  of  the  Fireground  Commander’s  job  (refer  to 
Appendix  B).  The  instructor  verbally  introduced  these  topics,  provided  time  for  the  students  to  read 
through  the  materials,  and  addressed  the  students’  questions.  Students  investigating  the  signal  threat 
analysis  task  of  the  Electronic  Warfare  (EW)  supervisors  were  provided  written  materials  describing 
the  domain  and  the  task,  and  a  list  of  terms  specific  to  the  EW  domain  (refer  to  Appendix  B).  A 
videotape  of  an  EW  technician  describing  the  signal  threat  analysis  task  was  presented. 
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Cognitive  Demands  Table 


What  is  the 
difficult  cognitive 
element? 

Why  is  it  difficult? 

What  cures  does  an  experienced 
driver  attend  to? 

What  strategies  does  an 
experienced  driver  use? 

Detecting  uneven 
wear. 

Novices  don’t  know 
what  it  is  supposed  to 
look  like. 

Novices  don’t  realize 
the  importance. 

Glance  at  the  tires  on 
a  regular  basis. 

Notice  a  shake  while 
driving. 

Novices  don’t  notice 
until  the  shake  is  very 
pronounced. 

Steering  wheel  shaking 

Steering  wheel  pulling  to  one  side 

Rhythmic  slapping  noises  coming 
from  the  tire. 

Play  with  the  steering, 
looking  for  other 

symptoms  (i.e.,  does  it 
shake  only  at  certain 
speeds?). 

Listen  for  noises  from  the 

tires. 

Spinning  tire  and 
feeling  for  bumps  on 
the  tire. 

Novices  don’t  know  to 

do  it. 

Tire  is  dirty  and 
rough — hard  to 

distinguish  rough  tire 
tread  from  real 

problem. 

Look  for  wobble  as  you  spin  the 
tire — broken  band. 

Run  hand  along  sidewall,  feel  for 
ridges — broken  band. 

Feel  for  bumps  along  the  tread  itself. 

Once  a  problem  is 
detected,  how  to 
diagnose  or  correct 
it? 

Novices  don’t 

understand  the 

connection  between 
the  wear  pattern  and 
what  is  going  on 
mechanically. 

Imagine  the  tire  is 
going  down  the  road — 
what  would  have  to  be 
happening  for  it  to  wear 
like  this? 

Figure  5.  Car  mechanic  Cognitive  Demands  Table  from  introductory  workshop. 


Physical  Skills 

Mental  Skills 

Communication 

Knowledge 

Administrative 

Skills 

Accomplish 

Load 

Achieve 

Communicate 

Administer 

Adjust 

Locate 

Analyze 

Define 

Coordinate 

Align 

Manipulate 

Calculate 

Describe 

Decide 

Apply 

Measure 

Choose 

Explain 

Deliver 

Balance 

Move 

Compare 

Express 

Draw 

Calibrate 

Operate 

Compute 

Identify 

Fill  out 

Change 

Perform 

Condense 

Illustrate 

Instruct 

Check 

Plot 

Decide 

List 

List 

Clean 

Position 

Derive 

Name 

Manage 

Complete 

Remove 

Determine 

State 

Report 

Construct 

Repair 

Diagnose 

Summarize 

Submit 

Correct 

Replace 

Distinguish 

Tell 

Deenergize 

Show 

Evaluate 

Write 

Demonstrate 

Start 

Interpret 

Employ 

Stop 

Monitor 

Energize 

Test 

Observe 

Enter 

Trace 

Recognize 

Exchange 

Troubleshoot 

Select 

Inspect 

Use 

Solve 

Install 

Utilize 

Synthesize 

Isolate 

Figure  6.  Learning  objectives  action  verbs. 


ACTA  Workshop 

The  purpose  of  the  ACTA  workshop  was  to  describe  the  ACTA  techniques  to  the  students  and 
to  provide  them  with  an  opportunity  to  practice  the  techniques  under  the  guidance  of  workshop 
facilitators  who  had  prior  experience  conducting  Cognitive  Task  Analysis  interviews.  The  first  part 
of  the  ACTA  workshop  provided  additional  background  information  on  Cognitive  Task  Analysis. 
This  background  information  included  an  example  of  a  behavioral  task  analysis  of  a  pilot 
conducting  a  take-off,  highlighting  the  cognitive  elements  of  the  task  that  were  presented  as 
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straight-forward  behaviors.  Following  the  presentation  of  the  background  material,  each  of  the  three 
ACTA  techniques  was  taught  as  described  in  the  following  paragraphs.  (The  techniques  themselves 
were  described  in  detail  earlier  in  this  report.) 

The  Task  Diagram  was  introduced  as  a  technique  that  is  used  to  obtain  an  overview  of  the  task; 
it  identifies  the  aspects  of  the  task  that  require  expertise  and  thus  frames  the  rest  of  the  Cognitive 
Task  Analysis.  The  workshop  leader  distributed  laminated  job  aids  that  described  the  steps  involved 
in  constructing  a  Task  Diagram  (Figure  7).  She  reviewed  these  steps  while  working  through  an 
example  of  a  Task  Diagram  based  on  an  interview  she  had  conducted  previously  with  an  auto 
mechanic.  Students  were  encouraged  to  ask  questions  throughout  this  demonstration.  Next,  the 
workshop  leader  instructed  the  students  to  follow  the  steps  printed  on  the  job  aid  while  she 
conducted  a  Task  Diagram  interview  with  one  of  the  workshop  facilitators.  Following  the 
demonstration,  the  students  were  once  again  encouraged  to  ask  questions.  At  this  point,  each  student 
was  paired  with  another  student  (or  with  a  workshop  facilitator  when  there  were  an  odd  number  of 
students  in  the  workshop)  and  instructed  to  interview  their  partners  using  the  Task  Diagram 
technique.  In  order  to  conduct  these  interviews,  each  student  had  to  think  of  a  domain  in  which  s/he 
was  an  expert  (e.g.,  cooking,  jogging,  etc.).  Workshop  facilitators  circulated  between  interview 
pairs  to  answer  questions  and  to  determine  whether  or  not  the  students  were  implementing  the  Task 
Diagram  technique  properly.  After  the  students  had  completed  their  Task  Diagrams,  the  workshop 
leader  facilitated  a  discussion  of  what  they  had  found  to  be  most  difficult  about  this  process,  and 
answered  any  additional  questions  raised  by  the  students. 

The  Knowledge  Audit  was  taught  following  the  same  general  procedures  described  above  for 
the  Task  Diagram.  After  a  brief  introduction  in  which  the  workshop  leader  reviewed  the  key  aspect 
of  expertise  (e.g.,  perceptual  skills,  recognition  of  anomalies,  improving,  etc.),  she  distributed  a 
laminated  job  aid  for  the  Knowledge  Audit  as  shown  in  Figure  8.  The  workshop  leader  walked  the 
students  through  an  example  of  a  Knowledge  Audit  conducted  with  an  auto  mechanic,  and  then 
demonstrated  the  Knowledge  Audit  using  a  different  example  with  a  workshop  facilitator.  In  both  of 
these  examples,  the  workshop  leader  demonstrated  the  link  between  the  Task  Diagram  and  the 
Knowledge  Audit.  That  is,  the  subtasks  that  had  been  circled  (to  indicate  they  involved  expertise) 
when  constructing  the  Task  Diagram  with  these  SMEs  were  then  explored  in  more  detail  by  using 
the  Knowledge  Audit  probes.  Following  the  demonstration  of  the  Knowledge  Audit,  students 
practiced  this  technique  with  their  partners  while  workshop  facilitators  circulated  among  the  pairs. 
After  having  a  chance  to  practice  the  Knowledge  Audit,  students  discussed  any  difficulties  they  had 
and  the  workshop  leader  answered  any  remaining  questions. 
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TD-8 

TD-9 

TD-10 

TD-11 


Application 


iagram.  Lists  the  procedures  of  a  task  in  a  linear  fashion. 


Go  into  this  interview  knowing  which  task  you  want  to  analyze.  You  will 
record  the  interviewee’s  responses  on  a  whiteboard  or  large  paper. 

Record  the  Task  of  Interest  at  the  top.  center  of  whiteboard. 

Ask  the  interviewee,  “Please  decompose  this  task  into  subtasks.  There  should 
be  at  least  three  sub-tasks,  but  no  more  than  six.” 

Record  each  Subtask  from  left  to  right  across  the  whiteboard. 

Ask  the  interviewee,  “Which  subtasks  require  the  most  expertise?” 

Place  circles  around  the  tasks  that  _ _ ^  Taskoffatoc^ 

require  the  most  expertise  and  (s^-4 Sut*** 

squares  around  the  rest  of  the  tasks.  v — s 


Record  the  first  Subtask  that  requires 
expertise  on  the  whiteboard. 


Ask  the  interviewee,  “Please  I - 1 

decompose  this  subtask  into  sub-sub  I - 

tasks.  Again,  there  should  be  at  least  three,  but  no  more  than  six.” 

Record  the  Sub-sub  tasks  on  the  whiteboard. 

Ask  the  interviewee,  “Which  of  these  sub  sub  tasks  require  the  most  expertise?’ 

Circle  those  that  require  expertise  and  place  squares  around  the  rest. 

Continue  decomposing  subtasks  until  you  have  a  diagram  for  each  one  that 
requires  expertise.  DO  NOT  decompose  sub-sub  tasks. 


Use  this  diagram  when  conducting  the  Knowledge  Audit  to  limit  the  interview 
to  those  tasks  that  require  expertise. 


1995.  Klein  Associates  Inc. 

Figure  7.  ACTA  Task  Diagram  Interview  Guide,  1995. 
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KA 


OWiedge  Audit.  Contrasts  what  experts  know  and  novices  don't 


Perceptual 

Skills 

Anomaly 


In  the  Task  Diagram  you  identified  the  sub*  and  sub-sub  tasks  that  require  the  most  expertise.  G<>  into  this 
interview  knowing  the  sub-tasks  you  want  to  analyze.  ’ _ „ _ 


Write  the  Task  of  Interest  at  lop.  center  of 
whiteboard-  Divide  the  remaining  space  into  three 
columns  with  headings  that  match  the  illustration  ort 
the  right. 


Example 


Task  of  interest 


Why  Difficult  Cues  &  Strategies 


i  Anomaly 

Use  the  probes  listed  below  to  elicit  examples  of  the  example  of  Anomaly 

various  aspects  of  expertise.  Record  the  first  example 
in  column  tine.  Ask  questions  KA-3  and  KA-f  before  1  Past  &  Future 
moving  on  to  the  next  probe.  Example. 

For  each  example,  ask,  *Why  is  this  task  hard  for  novices  or  why  don't  novices  know  to  do  that?”  Record 
answers  in  middle  column  under  the  heading  Why  Difficult 

For  each  example,  ask,  "What  cues  or  strategies  do  you  use  in  this  situation  T*  Record  answers  in  third  column 
under  Cues  and  Strategies. 


Knowledge  Audit  Probes 

Experts  detect  cues  and  patterns  and  make  discriminations  that  novices  can't  sec.  Can  you  think  or  any  examples 
here? 

Experts  can  notice  when  something  unusual  happens.  They  can  quickly  detect  deviations.  They  also  notice  when 
something  that  should  happen  doesn't.  Is  this  true  here?  Can  you  give  me  an  example? 

Experts  can  guess  how  the  current  situation  arose  and  they  can  anticipate  how  the  current  situation  will  evolve. 
Can  you  think  of  any  instance  in  which  this  happened,  cither  where  experts  were  successful  or  novices  fell  short? 


■  Big  Picture 

■  Tricks  of  the 
Trade 


If  you  were  watching  novices,  how  would  you  know  that  they  don't  have  the  big  picture? 
Are  there  tricks  of  the  trade  that  you  use? 


I  Improvising 
or  Noticing 
Opportunities 

■  Self- 

monitoring  & 
Adjustment 


Can  you  recall  a  situation  when  you  noticed  that  following  the  standard  procedure  wouldn't  work?  What  did  you 
do?  Can  you  think  of  an  example  where  the  procedure  would  have  worked  but  you  saw  that  you  could  get  more 
from  the  situation  by  taking  a  different  action? 

Experts  notice  when  their  performance  is  sub-par,  and  can  often  figure  out  WHY  that  is  happening  (e.g..  high 
workload,  fatigue,  boredom,  distraction)  in  order  to  make  adjustments.  Can  you  think  of  any  examples  where 
you  did  this? 


Equipment 


Scenario 
from  Hell 


Unless  you're  careful,  the  equipment  can  mislead  you.  Novices  usually  believe  whatever  the  equipment  says. 

Can  you  think  of  examples  where  you  had  to  rely  on  experience  to  avoid  being  fooled  by  the  equipment? 

If  you  were  going  to  give  someone  a  scenario  to  teach  someone  humility— that  this  is  a  tough  job-what  would  you 
pul  into  that  scenario?  Did  you  ever  have  an  experience  that  taught  you  humility  in  performing  this  job? 


Figure  8.  ACTA  Knowledge  Audit  Interview  Guide,  1995. 
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The  Simulation  Interview  was  the  final  technique  introduced  to  the  students.  After  explaining 
that  the  Simulation  Interview  was  used  to  highlight  cognitive  elements  of  a  task  within  the  context 
of  a  specific  incident,  the  workshop  leader  distributed  the  job  aid  shown  in  Figure  9  and  a  one-page 
written  simulation  that  had  been  given  to  an  auto  mechanic.  She  reviewed  how  to  conduct  a 
simulation  interview  using  the  auto  mechanic  example.  Because  it  was  not  feasible  to  develop 
written  simulations  for  each  domain  of  expertise  with  which  the  students  had  conducted  their  Task 
Diagrams  and  Knowledge  Audits,  the  workshop  leader  coached  the  students  as  they  demonstrated 
the  simulation  interview  with  a  workshop  facilitator.  For  this  demonstration,  a  one-page  written 
simulation  about  baking  cookies  was  distributed.  The  workshop  leader  facilitated  a  group 
discussion  in  which  the  decision  points  in  the  scenario  were  identified.  Next,  the  workshop  leader 
demonstrated  how  to  use  the  Simulation  Interview  probes  for  the  first  decision  point,  and  then  she 
selected  individual  students  to  practice  this  technique  for  other  decision  points  in  the  scenario.  The 
workshop  facilitators  provided  feedback  to  the  students.  Following  the  demonstration,  the  workshop 
leader  answered  any  remaining  questions  about  the  Simulation  Interview  technique  and  any  other 
questions  the  students  had  about  the  other  ACTA  techniques. 

Other  Workshops  Conducted 

In  addition  to  the  workshops  developed  for  training  the  participants  in  the  Evaluation  Study, 
additional  workshops  were  conducted  by  Klein  Associates  as  part  of  this  research  effort.  Some  of 
these  workshops  were  presented  at  professional  conferences,  while  others  were  presented  to  Navy 
personnel.  These  workshops  are  briefly  summarized  below. 

Professional  Conferences 

ACTA  workshops  were  presented  at  the  annual  meetings  of  the  Human  Factors  and  Ergonomics 
Society  in  1995  and  1996.  At  the  1995  meeting,  we  presented  a  half-day  workshop  that  was  so  well 
received  we  decided  to  expand  the  workshop  to  a  full-day  format  for  the  1996  meeting.  In  both 
workshops,  we  introduced  and  demonstrated  the  three  ACTA  interview  tools  (i.e.,  the  Task 
Diagram,  the  Knowledge  Audit,  and  the  Simulation  Interview)  and  allowed  a  limited  amount  of 
time  for  participants  to  practice  with  these  tools.  In  addition,  these  workshops  included  a  review  of 
other  Cognitive  Task  Analysis  techniques  and  concluded  with  a  discussion  of  how  to  apply  the 
ACTA  tools  to  specific  problem  domains  of  interest  to  the  workshop  participants. 

During  the  course  of  this  research  project,  we  also  made  several  presentations  to  introduce  the 
ACTA  tools  to  professionals  who  might  find  them  useful  for  their  jobs.  We  gave  a  presentation  in 
April,  1995,  at  the  Southwestern  Ohio  Chapter  of  the  American  Society  for  Training  and 
Development  in  which  we  described  the  ACTA  tools  and  demonstrated  them  with  the  help  of  a 
local  firefighter  who  served  as  the  SME.  We  also  gave  a  presentation  to  the  San  Diego  Chapter  of 
the  International  Society  for  Performance  and  Improvement  in  October,  1996. 
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Simulation  Interview.  Highlights  the  cognitive  dements  of  a  task. 

Obtain  a  simulation  of  the  task.  The  simulation  does  not  have  to  be  high  fidelity;  it  can  be  a 
paper  and  pencil  simulation,  video,  or  whatever  is  available. 

Ask  the  SME,  “Please  review  the  simulation  keeping  in  mind  that  I  will  be  asking  you  about 
the  decisions  and  judgments  you  would  have  made  in  this  situation.”  Offer  the  SME  pencil 
and  paper  on  which  to  keep  notes.  _  ■ _ 

Events 

.  Decisions  Situation  Critical  Potential 

DlVtde  a  whiteboard  into  6  Judgments  Assessment  Actions  Cues  Alternatives  Errors 

columns  with  headings  that  Evgnt  ~ 

match  the  illustration  on  the 

.  t  Even/  $2 

nght 

Bvmnt #3 

After  the  SME  has  reviewed 

the  simulation,  ask:  ‘Think  back  over  the  scenario.  Please  list  the  major 
events/judgments/decision  points  that  occurred  during  the  incident.  As  you  name  them,  I  am 
going  to  list  them  in  the  left  column  on  the  board.” 

For  each  event  in  the  left  column,  ask  the  questions  listed  below.  Ask  all  five  questions  about 
a  specific  event  before  moving  on  to  the  next  event.  Record  the  answers  to  each  question  in 
the  appropriate  column. 


■  Situation  What  do  you  think  is  going  on  here?  What  is  your  assessment  of  the  situation  at  this  point  in 
Assessment  time? 

■  Actions  What  actions,  if  any,  would  you  take  at  this  point  in  time? 


■  Critical  What  pieces  of  information  led  you  to  this  situation  assessment/action? 

Cues 

■Alternatives  Are  there  any  alternative  ways  you  could  interpret  this  situation?  Are  there  any  alternative 
courses  of  action  that  you  would  consider  at  this  point? 


Potential  What  errors  would  an  inexperienced  person  be  likely  to  make?  Are  there  cues  they  would 
Errors  miss? 
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We  asked  the  professionals  attending  both  of  these  meetings  to  provide  us  with  feedback  about 
the  relevance  and  usefulness  of  the  ACTA  tools. 

Navy  Personnel 

We  conducted  workshops  with  Navy  personnel  as  part  of  our  user  analysis.  Four  different 
workshops  were  presented  to  Navy  personnel  to  determine  their  reactions  to  the  ACTA  tools.  In 
June,  1995,  one  workshop  was  presented  to  several  different  groups  of  Navy  personnel;  this 
workshop  included  researchers  from  the  Navy  Personnel  Research  and  Development  Center 
(NPRDC),  instructors  from  the  Antisubmarine  Tactical  Air  Control  School,  and  instructors  from  the 
EW  School.  Additional  workshops  were  presented  to  instructors  from  the  Physician  Assistant 
School  and  the  Air  Intercept  Controller  School  in  March,  1995.  Another  workshop  was  given  to 
personnel  working  at  the  Naval  School  of  Health  Sciences  in  San  Diego  in  January,  1996.  In  each  of 
these  workshops  we  described  and  demonstrated  the  different  ACTA  tools  and  then  allowed  time 
for  the  participants  to  practice  with  the  ACTA  tools.  We  asked  the  participants  to  provide  us  with 
feedback  about  the  usability  of  the  tools  and  the  relevance  of  the  tools  to  their  Navy  job 
requirements. 

Additionally,  a  workshop  depicting  an  early  version  of  the  ACTA  techniques  was  presented  to 
representatives  from  the  Chief  of  Navy  Education  and  Training  in  Pensacola,  FL  in  February,  1995. 
A  follow-up  presentation  to  describe  a  further  refined  set  of  ACTA  tools  was  conducted  in  April, 
1996,  with  representatives  from  both  Chief  of  Naval  Education  and  Training  (CNET)  and  the  Naval 
Education  and  Training  Program  Management  Support  Activity  (NETPMSA). 

Lessons  Learned 

As  part  of  this  research  project,  Klein  Associates  conducted  over  a  dozen  workshops  in  which 
we  trained  a  wide  variety  of  individuals  in  how  to  use  the  ACTA  tools.  We  continually  modified 
and  improved  the  workshops  based  on  feedback  from  participants  and  on  our  own  observations 
regarding  the  effectiveness  of  the  training  methods  employed.  The  lessons  we  learned  from 
conducting  these  workshops  are  summarized  below. 

It  is  important  for  the  workshop  participants  to  understand  the  purpose  of  using  the  ACTA  tools. 
In  order  for  them  to  understand  what  they  will  gain  from  using  these  particular  interview  tools,  they 
must  first  understand  something  about  the  nature  of  expertise  and  the  importance  of  the  cognitive 
aspects  of  the  task  they  are  studying.  Initially,  we  spent  relatively  little  time  in  the  workshops 
discussing  the  nature  of  expertise  and  the  importance  of  cognition.  We  now  believe  this  is  a  critical 
first  step  in  training  people  how  to  use  the  ACTA  tools. 

We  always  recognized  the  need  for  participants  to  practice  with  the  ACTA  tools.  However 
initially,  we  did  not  allocate  very  much  time  in  the  workshops  for  helping  the  participants  select  a 
task  that  they  could  use  for  this  practice;  we  simply  instructed  the  participants  to  think  of  any  task 
for  which  they  had  some  expertise.  In  several  cases  we  noticed  that  the  participants  were  struggling 
to  apply  the  ACTA  tools  to  tasks  that  were  almost  totally  proceduralized  and  did  not  involve  any 
significant  amount  of  cognitive  elements.  It  is  much  easier  to  practice  the  ACTA  tools  with  tasks 
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that  involve  more  cognitive  elements.  We  found  that  another  benefit  of  spending  more  time 
discussing  the  nature  of  expertise  and  the  importance  of  cognition  at  the  beginning  of  the  workshop 
was  that  this  discussion  helped  participants  select  an  appropriate  task  for  practicing  the  ACTA  tools. 

We  used  very  simplistic  tasks  (e.g.,  cookie  baking)  for  demonstrating  the  ACTA  tools  to  the 
workshop  participants.  We  were  careful  to  select  simple  tasks  that  involved  a  sufficient  number  of 
cognitive  elements.  This  approach  seemed  to  be  very  effective  for  training  participants  in  how  to 
use  the  ACTA  tools.  We  were  concerned  that  if  we  used  a  more  complex  task  the  participants  would 
be  struggling  to  understand  an  unfamiliar  domain,  and  not  be  able  to  focus  on  the  ACTA  methods. 
The  drawback  to  using  simplistic  tasks  is  that  these  tasks  are  not  representative  of  the  types  of  tasks 
that  a  professional  would  actually  study  using  the  ACTA  tools.  We  believe  the  best  way  to  conduct 
these  workshops  is  to  use  simplistic  tasks  for  demonstration  purposes  early  in  the  workshop,  and 
then  to  conclude  the  workshops  with  a  discussion  of  the  application  of  the  ACTA  tools  to  problem 
domains  of  interest  to  the  participants.  For  these  discussions,  we  would  ask  participants  for 
examples  of  domains  that  they  would  like  to  study  with  the  ACTA  tools  and  then  have  the  entire 
group  discuss  how  to  apply  the  tools  to  these  particular  domains.  We  found  this  technique  to  be 
very  effective  for  helping  participants  learn  how  to  use  the  ACTA  tools  with  more  complex  tasks. 

The  workshops  we  conducted  varied  in  many  ways:  some  lasted  2  hours,  some  lasted  6  hours, 
the  number  of  participants  ranged  from  four  to  50,  some  workshops  included  a  very  homogenous 
group  of  participants  (e.g.,  all  students,  all  physician  assistants),  whereas  others  included 
participants  representing  several  different  professions.  Based  on  our  experience,  the  most  effective 
workshops  lasted  approximately  6  hours  and  were  conducted  with  groups  of  10-12  participants  from 
a  variety  of  different  backgrounds.  A  heterogeneous  group  of  participants  provides  the  opportunity 
to  learn  about  a  wider  variety  of  tasks  to  which  the  ACTA  tools  can  be  applied.  A  6-hour  format 
allows  sufficient  time  for  participants  to  practice  the  methods  and  receive  feedback  from  the 
workshop  facilitators  who  are  experienced  using  the  ACTA  tools.  Ideally,  there  would  be  a 
sufficient  number  of  workshop  facilitators  so  that  each  pair  of  participants  could  receive 
individualized  feedback  throughout  the  practice  sessions.  However,  this  typically  is  not  feasible  due 
to  the  cost  involved,  and  the  workshops  can  be  mn  successfully  with  two  facilitators  who  circulate 
among  the  pairs  of  participants  while  they  practice  using  the  ACTA  tools. 

After  conducting  ACTA  workshops  with  a  wide  variety  of  participants  it  has  become  clear  to  us 
that  there  are  large  individual  differences  in  the  ease  with  which  people  learn  how  to  use  the  ACTA 
tools.  Some  participants  are  able  to  quickly  master  these  tools;  whereas,  others  require  a  great  deal 
of  individualized  feedback  before  they  are  able  to  use  the  tools  effectively. 


Evaluation  Study 

The  primary  goal  of  the  project  was  straightforward:  develop  efficient  and  effective  Cognitive 
Task  Analysis  tools  for  use  by  Instructional  Designers  and  others  outside  of  the  research 
community.  How  to  test  the  ACTA  tools  in  order  to  evaluate  their  usability,  validity,  and  reliability 
was  a  more-complex  issue,  and  one  for  which  relatively  little  guidance  exists  within  the  current 
research  literature.  We  found  ourselves  in  the  position  of  developing  measures  and  methods  of 
evaluation  as  part  of  the  evaluation  effort  itself. 

As  Hoffman,  Crandall,  and  Shadbolt  (in  preparation)  point  out,  the  question  of  how  to 
empirically  verify  a  knowledge  base,  and  the  methodologies  used  to  articulate  and  represent  that 
knowledge  base,  has  received  little  attention  from  the  research  community.  Many  Cognitive  Task 
Analysis  methods  are  evaluated  solely  on  the  basis  of  subjective  judgments  of  whether  or  not  they 
seemed  to  work  for  a  particular  application  or  project.  Exceptions  include  work  by  Crandall  and  her 
colleagues  assessing  the  validity  and  reliability  of  data-gathering  skills  using  the  Critical  Decision 
method  (Crandall  &  Calderwood,  1989;  Crandall  &  Gamblian,  1991;  Crandall  &  Getchell-Reiter, 
1993;  Taynor,  Crandall  &  Wiggins,  1987),  and  method  comparisons  by  Hoffman  and  colleagues 
(Hoffman,  1987;  Hoffman,  Shadbolt,  Burton  &  Klein,  1995).  The  evaluation  study  described  in  this 
section  not  only  attempts  to  address  issues  of  validity  and  reliability  for  a  specific  set  of  Cognitive 
Task  Analysis  techniques,  but  also  addresses  a  number  of  issues  that  surround  the  assessment  of 
validity  and  reliability  within  the  context  of  real-world  tasks. 

Methods 

An  evaluation  of  the  ACTA  techniques  was  conducted  to  establish  the  validity  and  reliability  of 
the  data  gathered  using  the  methods,  and  to  assess  the  usability  of  the  techniques.  In  addition,  a 
comparison  of  information  gathered  using  ACTA  techniques  to  data  gathered  using  unstructured 
interview  techniques  was  conducted.2 

Parallel  studies  were  conducted  in  two  domains  for  this  evaluation.  Our  intention  was  to  test  the 
ACTA  techniques  with  a  sample  of  naive  users  —  people  who  lacked  knowledge  or  experience  with 
Cognitive  Task  Analysis  or  Instructional  Design.  A  novice  sample  would  allow  a  cleaner 
examination  of  the  impact  of  the  ACTA  methods  on  the  kind  and  quality  of  data  produced. 
Therefore,  students  from  graduate  programs  in  clinical,  human  factors,  or  cognitive  psychology 
were  recruited  via  postings  on  college  bulletin  boards  and  email,  to  conduct  interviews  and  generate 
instructional  materials  in  either  the  firefighting  domain  or  the  Electronic  Warfare  (EW)  domain. 
Volunteers  were  screened  to  make  sure  they  had  no  previous  knowledge  of  the  domain  they  would 
be  investigating,  no  previous  experience  conducting  Cognitive  Task  Analysis,  and  no  extensive 
experience  or  training  in  developing  course  materials.  Each  student  was  paid  $250  for  participation 


2 

One  set  of  Critical  Decision  method  interviews  was 
C.3  for  a  description  of  this  procedure. 


also  conducted  to  be  used  as  criterion 


data.  See  Appendix  C, 
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in  the  project.  Twelve  students  conducted  interviews  in  the  firefighting  domain  and  1 1  in  the  EW 
domain.3  The  SMEs  interviewed  were  experienced  Fireground  Commanders  from  the  greater 
Dayton  area  in  Dayton,  Ohio,  who  had  at  least  10  years  of  experience  including  experience  with 
fireground  command  and  were  recommended  by  the  Fire  Chief  of  each  local  fire  department;  and 
experienced  EW  technicians  from  Fleet  Training  Center  Pacific  in  San  Diego,  California,  who  had 
at  least  6  years  of  experience  as  EW  technicians,  including  4  years  at  sea  and  experience  as  an  EW 
supervisor.  (For  a  complete  description  of  participant  demographics,  see  Appendix  C,  C.l.) 

Within  each  domain,  students  were  placed  in  one  of  two  groups.  An  attempt  was  made  to  match 
the  groups  on  age,  gender,  and  education  level.  After  matching  the  students  on  these  criteria,  they 
were  randomly  assigned  to  groups.  All  students  attended  a  2-hour  workshop  introducing  the 
concepts  underlying  Cognitive  Task  Analysis,  describing  the  application  of  Cognitive  Task 
Analysis  to  the  development  of  instructional  materials,  and  providing  a  brief  overview  of  the 
domain  and  specific  task  they  would  be  investigating.  They  also  received  instruction  regarding  the 
training  materials  they  would  be  asked  to  develop  following  their  interviews  with  SMEs.  These 
materials  included  a  Cognitive  Demands  Table,  learning  objectives,  and  modifications  to  a  training 
manual.  (Workshop  materials  are  included  in  Appendix  B.) 

After  the  initial  2-hour  workshop,  the  matched  groups  of  students  were  separated  for  the 
remainder  of  the  study.  One  group,  referred  to  as  the  Unstructured  group,  was  provided  with 
instructions  to  conduct  interviews  with  SMEs  in  whatever  format  they  believed  would  be  most 
useful  for  gathering  cognitive  information.  They  were  told  to  spend  time  preparing  questions,  but 
were  not  given  any  direction  regarding  how  to  structure  the  interviews  or  specific  types  of  questions 
to  ask.  The  other  group,  referred  to  as  the  ACTA  group,  was  provided  with  a  6-hour  workshop  on 
the  ACTA  techniques,  including  knowledge  elicitation  and  knowledge  representation.  (Workshop 
materials  are  included  in  Appendix  B.) 

Students  in  both  the  Unstructured  and  ACTA  groups  then  participated  in  two  interviews  with 
SMEs.  Each  student  led  one  interview  with  an  SME  and  observed  an  interview  conducted  by 
another  student  with  another  SME,  and  thus  had  access  to  data  from  two  interviews.  Students 
working  in  the  firefighting  domain  were  asked  to  focus  on  the  size-up  task.  Students  working  in  the 
EW  domain  were  asked  to  focus  on  signal  threat  analysis.  All  interviews  were  scheduled  for  a  3- 
hour  block  of  time. 


3 

For  the  firefighter  study,  13  graduate  students  in  Dayton,  Ohio  were  recruited.  One  student  served  as  an  alternate 
for  the  ACTA  group.  This  student  completed  all  the  workshops,  but  did  not  actually  conduct  or  attend  any  interviews. 
For  the  EW  study,  11  graduate  students  in  San  Diego,  California  were  recruited  initially.  Due  to  illness  or  school 
commitments,  three  of  these  students  dropped  out  before  completion  of  the  workshops.  To  increase  the  number  of 
participants,  four  additional  students  were  recruited  and  another  set  of  workshops  was  conducted.  One  of  these  students 
served  as  an  alternate.  The  other  three  conducted  interviews  and  generated  instructional  materials  using  the  same 
format  and  procedures  as  those  initially  recruited. 
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Within  a  week  of  completing  the  two  interviews,  each  student  attended  a  4-hour  session  to 
analyze  the  data  and  develop  training  materials.  The  students  were  instructed  not  to  collaborate  or 
do  any  additional  work  with  their  interview  notes  prior  to  the  scheduled  session.  During  the  4-hour 
session,  they  were  required  to  structure  and  represent  the  information  obtained  in  interviews.  They 
were  provided  materials  and  instructions  and  asked  to: 

•  Consolidate  the  data  from  the  interview  using  the  Cognitive  Demands  Table  format. 

•  Develop  at  least  10  cognitive  learning  objectives  for  a  hypothetical  course  in  that  domain. 

•  Revise  or  add  to  training  manuals  (these  were  provided),  based  on  what  they  had  learned  in 
the  interviews. 

•  Complete  a  questionnaire  about  participation  in  the  study. 

In  addition,  all  students  who  had  been  exposed  to  the  ACTA  techniques  were  asked  to  fill  out  an 
ACTA  usability  questionnaire. 

Data  Transformation 

In  order  to  generate  quantitative  measure  of  utility  and  validity,  the  information  generated  by  the 
ACTA  tools  required  extensive  data  codification  and  transformation.  All  materials  generated  by  the 
sample  of  graduate  students  were  assessed  by  SMEs  and/or  cognitive  psychologists.  Wherever 
possible,  data  evaluation  was  carried  out  by  multiple  coders,  so  that  inter-rater  reliability  could  be 
assessed.  In  some  cases,  due  to  lack  of  availability  or  resource  constraints,  only  one  SME  was 
available  to  code  the  data.  Measures  were  devised  to  address  two  aspects  of  validity:  (1)  whether  the 
ACTA  tools  produced  information  that  was  predominantly  cognitive  in  nature,  and  (2)  whether  the 
information  produced  was  domain-specific  and  relevant.  Data  transformation  procedures  and 
associated  measures  are  described  in  detail  in  the  following  sections. 

Validity  Indices:  Cognitive  Demands  Table 

All  items  included  in  the  Cognitive  Demands  Tables  were  coded  by  two  Klein  Associates 
researchers,  Laura  Militello  and  Dr.  Rebecca  Pliske,  blind  to  treatment  group  (ACTA  vs. 
Unstructured),  for  whether  they  contained  cognitive  content.  The  criterion  for  inclusion  in  the 
cognitive  demand  category  was  that  the  item  addressed  a  cognitive  skill  or  a  cognitive  challenge 
that  a  firefighter/EW  operator  encounters  (e.g.,  “deciding  whether  or  not  water  supply  on  the  scene 
will  be  adequate,”  “devising  a  strategy  to  successfully  remove  people  from  a  burning  building”). 
Items  in  the  Noncognitive  demands  category  typically  referred  to  declarative  knowledge  that  the 
firefighter/EW  operator  should  have  (e.g.,  “know  the  initial  command  sequence”)  or  behaviors  (e.g., 
“return  resources”). 

In  order  to  establish  whether  students  using  the  ACTA  techniques  could  consistently  elicit 
information  across  relevant  cognitive  categories  (as  opposed  to  task-based  categories),  we 
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developed  a  coding  scheme  based  on  Rasmussen,  Pejterson,  and  Goodstein’s  (1994)  model  of 
decision  making.  The  categories  included  information  collection,  situation  analysis,  diagnosis, 
prediction,  value  judgment,  choice,  planning,  and  scheduling  (see  Appendix  C,  C.2,  for  category 
definitions).  Two  raters,  blind  to  the  students’  interview  group  (ACTA  vs.  Unstructured), 
independently  rated  30  percent  of  the  data.  The  raters  established  acceptable  inter-rater  agreement 
(%  agreement  =  74%).  The  rest  of  the  data  were  then  divided  among  the  two  raters  to  complete  the 
rating. 

Evaluation  of  the  domain-specific  content  of  the  Cognitive  Demands  Tables  was  based  on  the 
firefighting  portion  of  the  database.  A  task-based  coding  scheme  specific  to  the  firefighting  domain 
was  developed.  Based  on  firefighting  manuals  made  available  to  us  by  the  National  Fire  Academy, 
Emmitsburg,  MD,  we  established  that  there  are  three  primary  subtasks  for  which  the  Fireground 
Commander  is  responsible:  size-up,  strategy/tactics,  and  management.  For  the  firefighter  data,  the 
coders  independently  assessed  the  content  of  each  Cognitive  Demands  Table  item  and  assigned  it  to 
one  of  these  three  categories.  The  coders  established  reliability  (%  agreement  =  81%)  on  40  percent 
of  the  data,4  and  one  researcher  then  coded  the  remainder  of  the  data. 

We  believed  that  it  was  also  important  to  have  the  data  evaluated  by  domain  experts,  in  order  to 
assess  data  quality  and  relevance.  An  assessment  of  the  firefighter  data  was  carried  out  by  an 
instructor  for  the  Incident  Command  course  at  the  Ohio  Fire  Academy.  He  had  more  than  10  years 
of  firefighting  experience,  had  served  as  a  Fireground  Commander,  and  is  currently  involved  in  the 
development  of  course  materials  for  the  firefighter  courses  taught  at  the  Ohio  Fire  Academy.  The 
EW  SME  was  a  retired  U.S.  Navy  Electronic  Warfare  technician  who  had  extensive  experience  as 
an  operator,  a  supervisor,  and  as  an  instructor.  The  SMEs  were  asked  to  indicate  what  percentage  of 
the  information  contained  in  each  Cognitive  Demands  Table  would  be  likely  to  be  known  only  by 
experienced  personnel.  In  addition,  the  SMEs  were  asked  to  indicate  the  percentage  of  information 
contained  in  each  Cognitive  Demands  Table  that  would  be  relevant  for  experienced,  highly  skilled 
personnel  (Fireground  Commander/EW  supervisor)  as  opposed  to  a  person  with  little  experience  on 
the  job  (firefighter/new  EW  operator).  Given  that  one  objective  of  the  ACTA  techniques  is  to  elicit 
experience-based  knowledge  (as  opposed  to  declarative  knowledge,  which  is  easily  captured  using 
other  traditional  techniques  and  disseminated  via  textbooks),  we  wanted  to  distinguish  information 
that  only  an  experienced  person  would  know  from  that  which  people  newly  released  from  training 
would  know.  Both  of  these  questions  were  intended  to  distinguish  between  information  reflective  of 
experience-based  knowledge  vs.  classroom  knowledge. 

Validity  Indices:  Instructional  Materials 

In  addition  to  the  firefighter  SME  described  above,  a  second  instructor  from  the  Ohio  Fire 
Academy  was  recruited  to  provide  ratings  of  the  instructional  materials  generated  by  the  students  in 


“No  standard  for  acceptable  inter-rater  agreement  exists  (Meister,  1985;  Pedhazur  &  Schmelkin,  1991).  However, 
agreement  ratings  exceeding  70  percent  are  generally  accepted  as  adequate  for  this  type  of  coding.  Subsets  of  data  were 
analyzed  until  an  acceptable  level  of  agreement  was  reached.  The  remaining  data  were  then  analyzed  by  one  coder. 
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the  firefighting  domain.  He  also  had  more  than  10  years  experience  as  a  firefighter,  had  served  as  a 
Fireground  Commander,  and  is  currently  involved  in  the  development  of  course  materials  at  the 
Ohio  Fire  Academy.  In  addition,  two  EW  instructors  from  the  Electronic  Warfare  “A”  School  in 
Pensacola,  Florida  were  recruited  to  rate  the  instructional  materials  generated  by  graduate  students 
working  in  the  EW  domain.  Both  Electronic  Warfare  SMEs  held  a  rank  of  E-6  or  above,  had  served 
as  an  EW  supervisor,  and  had  experience  as  an  instructor  at  the  EW  “A”  School. 

Working  independently,  the  SMEs  in  each  domain  were  asked  to  evaluate  each  learning 
objective  and  training  manual  modification  for  accuracy,  importance,  and  whether  or  not  it  was 
currently  included  in  the  typical  firefighter  training/EW  instructional  program.  In  the  firefighting 
domain,  acceptable  inter-rater  agreement  was  obtained  for  the  accuracy  and  importance  ratings,  but 
not  for  the  rating  of  whether  or  not  the  information  described  in  the  learning  objective  was  currently 
covered  in  the  typical  firefighter  training  course.5  For  the  learning  objectives,  the  percent  agreement 
for  the  firefighter  SMEs’  accuracy  judgments  was  87.8  percent;  the  percent  agreement  for  the 
firefighter  SME’s  importance  ratings  was  71.4  percent.  For  the  modifications  to  the  student  manual, 
percent  agreement  for  accuracy  was  90.1  percent  and  for  importance  was  76.1  percent.  The 
accuracy  and  importance  ratings  made  by  the  SME  who  had  more  extensive  experience  in 
developing  training  materials  for  Fireground  Commanders  were  used  in  further  analyses. 

The  SMEs  in  the  EW  domain  were  not  able  to  reach  acceptable  inter-rater  agreement.  For  the 
learning  objectives,  percent  agreement  for  importance  was  58.5  percent  and  for  accuracy  it  was  67.9 
percent.  For  the  modifications  to  the  student  manual,  percent  agreement  for  importance  was  34.2 
percent  and  for  accuracy  it  was  61.7  percent.  Discussion  with  the  SMEs  revealed  that  depending 
upon  the  type  of  ship  one  serves  on,  the  EW  job  may  be  very  different.  Our  two  SMEs  had  served 
on  different  types  of  ships  and  were  currently  teaching  veiy  different  courses  (basic  tactics  vs. 
introductory  course  on  equipment).  As  a  result,  they  had  quite  different  perspectives  on  what  is 
important  for  an  EW  operator  to  learn  in  school  to  prepare  him/her  for  the  job.  For  all  further 
analyses,  we  used  the  ratings  from  the  SME  with  the  most  recent  and  most  extensive  sea 
experience. 

Results 

The  results  section  first  presents  our  findings  as  they  relate  to  the  usability,  validity,  and 
reliability  of  the  ACTA  techniques,  as  these  were  the  primary  questions  to  be  answered  by  our 
evaluation  study.  Thus  the  data  presented  in  the  following  sections  are  based  only  on  students  who 
completed  the  ACTA  workshops  and  used  these  methods  to  conduct  an  interview  with  a  SME.  The 
final  portion  of  the  results  section  discusses  the  data  as  it  relates  to  differences  between  the 
materials  generated  by  students  who  conducted  interviews  using  ACTA  vs.  those  students  who 


5Due  to  unacceptable  reliability  ratings,  no  further  analyses  were  conducted  on  data  relating  to  whether  the 
information  was  currently  covered  in  a  course.  Discussion  with  the  firefighter  SMEs  revealed  that  they  had  experience 
teaching  different  courses  and  therefore  had  different  perspectives  on  what  was  “typically”  included  in  firefighter 
instruction. 
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conducted  unstructured  cognitive  interviews.  Although  few  group  differences  were  found,  a 
discussion  of  how  large  intra-group  variability  impacted  this  study  is  presented. 

Usability 

In  evaluating  the  ACTA  tools,  we  were  interested  in  understanding  the  subjective  experiences  of 
both  the  interviewers  and  the  interviewees.  User  acceptance  is  key  to  the  success  of  this  project.  To 
assess  user  acceptance,  three  questionnaires  were  administered:  a  usability  questionnaire  focusing 
specifically  on  the  ACTA  techniques,  an  interviewee  questionnaire  eliciting  information  from  the 
SME’s  perspective,  and  an  interviewer  questionnaire  addressing  the  experience  of  participating  in 
all  aspects  of  the  study.  The  findings  from  the  questionnaire  data  are  presented  in  this  section. 

Usability  Questionnaire.  A  usability  questionnaire  was  administered  to  all  graduate  students 
who  used  the  ACTA  techniques.  Overall,  ratings  were  very  positive.  All  of  the  tools  were  rated  as 
useful.6  Mean  ratings  on  all  dimensions  were  above  “3”  on  a  5-point  scale,  where  “5”  is  the  most 
positive  rating  and  “1”  is  the  least  positive  rating  (see  Table  1).  These  data  indicate  that  graduate 
students  found: 

•  The  methods  to  be  easy  to  use. 

•  The  interview  guides  and  job  aids  to  be  flexible. 

•  The  output  of  the  interviews  to  be  clear. 

•  The  knowledge  representations  to  be  useful. 

Interviewee  Questionnaire.  Each  SME  was  asked  to  fill  out  a  brief  questionnaire  at  the  end  of 
the  interview.  If  the  ACTA  tools  are  to  be  accepted  in  an  operational  community,  the  impressions  of 
the  people  who  are  interviewed  will  have  considerable  influence.  If  the  SMEs  find  the  interview 
process  aversive,  or  do  not  find  that  they  are  given  an  opportunity  to  communicate  the  critical 
elements  of  the  job,  acceptance  of  the  ACTA  tools  will  be  greatly  compromised  within  an 
organization. 

The  questionnaire  data  indicate  that  the  interviewees  found  the  interview  experience  to  be 
pleasant  and  worthwhile.  Table  2  presents  the  means  for  each  question  for  those  SMEs  who 
participated  in  ACTA  interviews.7 


6A  three-way,  mixed  design  ANOVA  taking  into  account  the  domain  (Firefighting,  EW),  the  ACTA  techniques 
(Task  Diagram,  Knowledge  Audit,  Simulation  Interview),  and  the  individual  questions  (Questions  2,  3, 4,  and  5)  on  the 
usability  questionnaire  showed  no  difference  in  the  usability  of  the  three  techniques,  F  (2,  18)  =  1.34,  p  =  .29,  or  in  the 
usability  of  the  techniques  across  domains,  F  (1,  9),  p  <  1. 

7A  three-way,  mixed  design  ANOVA  taking  into  account  domain  (Firefighting,  EW),  interview  type  (ACTA, 
Unstructured),  and  question  (5  questions  from  questionnaire)  showed  no  domain  differences  in  the  interviewees’ 
responses,  F  (1,  20)  =  .82,  p<  1. 
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Table  1 

Usability  Questionnaire  Means  for  the  Graduate  Students  who 
Conducted  Interviews  Using  the  ACTA  Techniques 
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Table  2 


SME  Interviewee  Questionnaire  Means  for  the  SMEs 
Interviewed  Using  the  ACTA  Techniques 


Firefighting 

N  =  6 

Electronic  Warfare 
JV  =  5 

Mean  Totals 

A  =11 

Overall,  I  found  the  interview  to  be  a  pleasant 

3.6 

3.97 

experience. 

■bh 

(1.14) 

(0.89) 

The  format  of  the  interview  allowed  me  to  describe 

■ESBi 

3.6 

■EBB 

my  expertise. 

■Emi 

(1.14) 

HK 

I  thought  the  interview  lasted  too  long.a 

4.00 

(1  =  strongly  agree,  5  =  strongly  disagree) 

HI 

(0.71) 

(0.63) 

Participating  in  the  interview  gave  me  new  insights 

3.67 

3.20 

into  the  cognitive  aspects  of  my  job. 

(1.03) 

(1.48) 

1  1 

I  think  the  cognitive  aspects  of  my  job  that  were 

4.33 

4.00 

4.17 

discussed  during  the  interview  are  important  things 
for  a  novice  to  learn. 

(0.52) 

(1.22) 

(0.87) 

Note.  All  questions  used  a  five  point  rating  scale.  Unless  otherwise  stated,  1  =  strongly  disagree,  5  =  strongly  agree. 
Standard  deviations  are  included  in  parentheses. 


“The  scale  for  Question  3  was  switched  so  that  a  high  score  indicates  a  more  positive  response  to  be  consistent  with 
other  questions  on  the  questionnaire. 

Interviewer  Questionnaire.  All  graduate  students  filled  out  a  questionnaire  at  the  completion 
of  their  participation  in  the  evaluation  study.  This  questionnaire  consisted  of  10  questions  intended 
to  capture  the  interviewer’s  subjective  experience,  addressing  issues  such  as  confidence  level, 
perceived  difficulty  or  ease  of  the  Cognitive  Task  Analysis  process,  etc.  The  means  for  each 
question  from  the  ACTA  group  are  presented  in  Table  3. 

These  data  indicate  that  graduate  students  in  both  domains  found  the  interviews  to  be 
informative  and  to  provide  cognitive  information  about  the  job  domain.  Based  on  the  information 
learned  via  ACTA  interviews,  the  graduate  students  found  the  development  of  a  Cognitive 
Demands  Table  and  the  generation  of  learning  objectives  to  be  easy.  Participants  indicated  that  they 
were  able  to  make  important  revisions  to  the  course  materials  provided.  Surprisingly,  participants 
responded  negatively  to  the  statement,  “I  want  to  conduct  more  interviews  because  I  still  want  more 
information.”  Our  only  explanation  for  this  is  that  because  our  participants  were  novices  to 
Cognitive  Task  Analysis,  they  did  not  anticipate  the  breadth  and  depth  of  knowledge  that  can  be 
gained  via  Cognitive  Task  Analysis  techniques. 
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Table  3 


Graduate  Student  Interviewer  Questionnaire  Means  for  the  Graduate  Students  who 
Conducted  Interviews  Using  the  ACTA  Techniques 


Firefighting 

A=6 

Electronic  Warfare 
N  =  5 

Mean  Totals 

A  =  11 

I  felt  confident  in  my  ability  to  lead  an  interview. 

1 

4.00 

■BOB 

■ 

(0.00) 

HHn 

I  learned  more  information  from  the  interview  I 

2.00 

3.00 

2.50 

observed,  than  from  the  one  I  led. 

(0.89) 

(1.23) 

(1.13) 

I  felt  I  had  sufficient  information  to  revise  the 

4.00 

3.60 

course  materials. 

(0.89) 

1 

(1.12) 

The  interviews  provided  me  with  important 

4.33 

3.60 

3.97 

information  about  the  cognitive  skills  involved  in 
this  job  domain. 

(0.52) 

(1.14) 

(0.89) 

I  wanted  to  conduct  more  interviews  because  I  still 

2.33 

mmmsm 

2.77 

wanted  more  information. 

(0.82) 

1 

(1.10) 

The  Cognitive  Demands  Table  was  easy  to  fill  out. 

■sal 

3.40 

■■ 

(0.89) 

■ 

It  was  easy  to  develop  course  objectives  based  on 

■■ 

3.20 

3.77 

information  specified  in  the  Cognitive  Demand 
Table. 

Hi 

(1.64) 

(1.25) 

I  was  able  to  use  the  information  to  make  important 

3.50 

3.40 

3.45 

changes  in  the  course  material. 

(1-14) 

(0.93) 

Overall,  I  found  the  interviews  to  be  informative. 

WMMm 

4.72 

HI 

(0.47) 

Given  the  information,  I  found  the  revision  of 

3.67 

3.00 

3.34 

course  materials  straightforward. 

(0.82) 

(1.23) 

(1.03) 

Note.  All  questions  used  a  five  point  rating  scale.  Unless  otherwise  stated,  1  =  strongly  disagree,  5  =  strongly 
agree.  Standard  deviations  are  included  in  parentheses. 


Validity 

Table  4  presents  data  that  addresses  three  central  questions  regarding  validity: 

•  Does  the  information  gathered  address  cognitive  issues? 

•  Does  the  information  gathered  deal  with  experience-based  knowledge  as  opposed  to 
declarative  knowledge? 

•  Do  the  instructional  materials  generated  contain  accurate  information  that  is  important  for 
novices  to  leam? 
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Table  4 


Quality  of  Outputs  for  Graduate  Students  Who  Conducted 
Interviews  Using  the  ACTA  Techniques 


Validity  Indicator 

Firefighter 

N  =  6 

Electronic 

Warfare 

N  =  5 

Percent  of  total  Cognitive  Demands  Table  items  coded  as  cognitive. 

92% 

94% 

Proportion  of  Cognitive  Demands  Table  information  experienced  personnel 

0.95 

I 

likely  to  know,  averaged  across  ACTA  users. 

(.05) 

1 

Proportion  of  Cognitive  Demands  Table  information  relevant  to  a 

0.87 

Fireground  Commander/EW  Supervisor,  averaged  across  ACTA  users. 

(0.10) 

Proportion  of  student  manual  modifications  rated  as  important  or  somewhat 

0.70 

wmam 

important,  averaged  across  ACTA  users. 

(.47) 

HUH 

Proportion  of  learning  objectives  rated  as  important  or  somewhat  important, 

■EH 

■a 

averaged  across  ACTA  users. 

HH 

■■ 

Proportion  of  student  manual  modifications  rated  as  accurate,  averaged 

msm 

across  ACTA  users. 

Proportion  of  learning  objectives  rated  as  accurate,  averaged  across  ACTA 

0.92 

users. 

(0.24) 

Note.  Standard  deviations  are  included  in  parentheses. 


The  extent  to  which  the  information  elicited  using  ACTA  was  cognitive  in  nature  was  assessed 
by  examining  every  item  contained  in  the  Cognitive  Demands  Tables  for  its  cognitive  content.  The 
cognitive  content  codings  indicate  that  fully  93  percent  of  the  items  generated  address  cognitive 
issues.  More  specifically,  in  the  firefighter  study,  92  percent  of  the  items  were  rated  as  cognitive 
and  in  the  EW  study  94  percent  of  the  cognitive  demand  items  generated  by  the  students  using 
ACTA  were  rated  as  cognitive. 

To  address  the  issue  of  whether  the  ACTA  tool  provided  a  means  to  elicit  experience-based 
knowledge,  SMEs  were  asked  to  make  a  global  assessment  of  each  Cognitive  Demand  Table  and  to 
assign  a  percentage  to  each  that  reflected  the  proportion  of  information  it  contained  that  only  highly 
experienced  personnel  would  be  likely  to  know.  The  inference  here  is  that  such  information  is 
reflective  of  experience-based,  as  opposed  to  declarative,  knowledge.  The  information  that  newly- 
trained  personnel  possess  is  more  likely  to  have  been  learned  in  a  classroom  setting  and  is  likely  to 
be  predominantly  declarative  knowledge. 

The  findings  offer  strong  support  that  the  ACTA  tools  allowed  students  to  elicit  important  and 
relevant  domain  information.  In  the  firefighter  study,  the  percentage  of  content  of  the  Cognitive 
Demands  Tables  that  was  judged  to  be  information  that  only  highly  experienced  personnel  would 
know  averaged  95  percent.  In  the  EW  domain,  the  same  assessment  yielded  an  average  of  90 
percent  across  the  ACTA  group.  In  response  to  questions  regarding  information  relevance,  a 
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substantial  percentage  of  information  in  the  Cognitive  Demands  Tables  (M  =  73%)  was  rated  as 
relevant  for  a  Fireground  Commander.  Similar  results  were  found  in  the  EW  domain  where  a  mean 
of  87  percent  of  the  information  in  the  Cognitive  Demands  Tables  was  rated  as  relevant  to  an  EW 
supervisor. 

Our  third  validity  question  focused  on  the  importance  and  accuracy  of  the  instructional  materials 
generated  by  ACTA  users.  Our  measures  included  ratings  of  importance  and  accuracy  by  domain 
experts.  The  3-point  importance  ratings  were  collapsed  into  a  dichotomy,  with  “important”  or 
“somewhat  important”  ratings  combined  into  a  single  importance  indicator.  Accuracy  had  been 
assessed  as  a  dichotomy  (accurate  vs.  not).  Findings  indicate  that  in  both  domains,  content  of 
instructional  materials  generated  by  ACTA  were  viewed  as  important  domain  information  for 
novices  to  learn.  In  the  firefighting  domain,  an  average  of  70  percent  of  the  instructional  material 
modifications  generated,  and  95  percent  of  the  learning  objectives  generated  by  each  student  were 
rated  as  important.  In  the  EW  domain,  these  averages  were  93  percent  and  83  percent,  respectively. 

Accuracy  evaluations  were  also  high,  particularly  for  the  firefighting  data.  In  the  firefighting 
domain  an  average  89  percent  of  the  modifications  to  the  student  manuals  were  rated  as  accurate 
and  92  percent  of  the  learning  objectives  were  rated  as  accurate.  In  the  EW  domains,  these  averages 
were  65  percent  (modifications  to  the  student  manual)  and  54  percent  (learning  objectives).  We 
suspect  that  the  lower  accuracy  ratings  in  the  EW  domain  were  due  to  the  more  technical  nature  of 
the  domain.  The  environment  in  which  an  EW  operator/supervisor  works  was  so  foreign  to  the 
graduate  students  that  understanding  and  using  the  terminology  and  acronyms  that  EW  operators 
/supervisors  use  to  describe  the  equipment  and  environment  presented  additional  difficulty.  There 
were  a  number  of  cases  in  which  the  EW  SMEs  rating  the  data  indicated  that  they  knew  what  the 
student  must  have  meant,  but  that  the  wording  used  was  incorrect. 

Reliability 

There  is  no  well-established  metric  or  method  for  assessing  the  reliability  of  Cognitive  Task 
Analysis  tools,  and  yet  the  issue  is  an  important  one.  Briefly,  the  question  is  whether  individuals 
using  a  particular  technique  are  able  to  generate  comparable  information,  so  that  the  tools  can  be 
considered  a  source  of  consistent  information,  given  the  same  (or  similar)  domain  expert  assessed  at 
different  points  in  time  and/or  by  different  knowledge  engineers.  This  is  a  much  simpler  matter 
when  one  is  dealing  with  highly-structured  interview  formats  or  scale  items  than  when  faced  with 
textual  knowledge  representations.  We  sought  to  address  the  issue  in  several  ways.  One  approach 
was  to  examine  whether  ACTA  users  consistently  elicited  the  same  types  of  cognitive  information. 
Therefore,  we  examined  the  content  of  the  cognitive  demands  generated  by  the  students,  to  see 
whether  they  had  generated  similar  information. 

One  set  of  analyses  examined  whether  ACTA  users  had  generated  similar  types  or  categories  of 
cognitive  information.  This  analysis  utilized  a  coding  scheme  based  on  Rasmussen,  et  al.,  (1994) 
model  of  decision  making  (found  in  Appendix  C,  C.2).  Each  item  in  each  Cognitive  Demands  Table 
was  coded.  In  the  firefighting  domain,  every  Cognitive  Demands  Table  (100%)  generated  by  the 
ACTA  group  contained  information  that  had  to  do  with  situation  analysis  and  planning.  All  but  one 
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of  the  Cognitive  Demands  Tables  (80%)  contained  data  on  information  collection.  Given  that 
students  were  instructed  to  focus  on  the  subtask  of  “size-up,”  which  consists  of  gathering  relevant 
information  in  order  to  accurately  assess  the  situation  and  develop  a  plan  of  action,  we  concluded 
that  students  in  this  study  were  able  to  consistently  elicit  relevant  cognitive  information  using  the 
ACTA  techniques.  Students  did  not  consistently  elicit  information  in  the  remaining  categories 
(diagnosis,  prediction,  value  judgment,  choice,  and  scheduling). 

The  same  analysis  was  carried  out  for  the  EW  study.  All  ACTA  users  generated  cognitive 
demands  that  included  information  about  situation  analysis  and  all  but  one  collected  data  in  the 
information  collection  category.  The  signal  threat  analysis  task  consists  primarily  of  gathering  the 
necessary  information  to  maintain  an  accurate,  current  assessment  of  the  situation  at  all  times. 
Again  the  data  indicate  that  the  students  consistently  elicited  relevant  cognitive  information  using 
the  ACTA  tools.  Data  across  the  two  domains  suggest  that  students  were  able  to  consistently  elicit 
comparable  cognitive  information  using  the  ACTA  techniques. 

A  second  coding  scheme,  specific  to  the  firefighting  data,  also  indicated  that  students 
consistently  elicited  similar  information.  The  Fireground  Commander  task  was  divided  into  three 
subtasks:  size-up,  strategy  and  tactics,  and  management.  All  the  ACTA  users  obtained  information 
in  each  of  these  categories.  The  bulk  of  the  information  gathered  focused  on  the  size-up  task  (62%), 
which  is  where  the  students  were  asked  to  focus  their  interviews.  Thus  we  conclude  that,  using 
ACTA,  people  were  consistently  able  to  get  important  cognitive  information  for  the  entire 
Fireground  Commander  task,  with  an  emphasis  on  the  size-up  task. 

We  also  attempted  to  assess  the  degree  of  overlap  of  specific  items  across  the  Cognitive 
Demands  Tables  generated  by  ACTA  users.  This  proved  extremely  difficult,  because  users  had  not 
been  constrained  in  level  of  detail,  phrasing,  or  specificity.  One  student  might  list  as  a  cognitive 
demand  “look  of  the  smoke”  while  another  noted  “color  and  movement  of  smoke.”  The  levels  of 
inference  required  by  raters  to  judge  the  degree  to  which  any  two  Cognitive  Demands  Table  items 
matched  were  similar,  or  were  different  became  unacceptable,  and  the  analysis  was  abandoned. 

However,  our  informal  examination  of  the  Cognitive  Demands  Tables  suggests  that  the  graduate 
students  did  not,  in  most  cases,  generate  identical  cognitive  demands.  This  is  not  surprising  given 
the  design  of  this  study.  In  order  to  reduce  intra-subject  variability,  we  excluded  from  the  study 
graduate  students  who  had  any  experience  in  the  domain  for  which  they  would  be  conducting  the 
Cognitive  Task  Analysis;  which  meant  that  all  of  the  students  were  working  at  a  disadvantage  in 
conducting  the  Cognitive  Task  Analysis.  When  we  describe  the  ACTA  tools  to  professional 
audiences,  we  recommend  that  time  is  spent  upfront  becoming  familiar  with  a  domain  before 
interviews  are  conducted.  In  this  case,  the  students  were  given  a  brief  overview  of  the  domain  and 
the  task  they  would  be  studying.  The  limited  time  we  had  with  the  graduate  students  did  not  allow 
for  the  recommended  level  of  familiarization  with  the  domain.  A  second  reason  why  the  graduate 
students  did  not  generate  the  same  cognitive  demands  is  that  each  student  was  exposed  to  only  two 
interviews  with  SMEs.  If  SME  availability  had  allowed  each  student  access  to  three  to  five  experts 
as  we  generally  recommend,  the  students  would  have  been  more  likely  to  have  heard  similar  things 
in  the  interviews. 
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Group  Differences 

One  of  the  drawbacks  of  the  evaluation  study  design  was  sample  size.  The  intensive  workshop 
preparation  necessary  to  train  subjects  in  ACTA  methods,  and  the  extensive  coding  and  data 
transformation  effort  necessary  to  provide  empirical  evaluation  data,  and  the  limited  number  of 
available  SMEs,  made  large  samples  simply  beyond  the  time  or  resources  available.  Obviously, 
with  the  small  group  sizes  we  had,  the  effects  associated  with  membership  in  the  ACTA  group  were 
going  to  have  to  be  very  strong  to  be  discemable  as  statistically  significant.  Nonetheless,  we  were 
surprised  to  find  so  few  differences  between  the  ACTA  group  and  the  unstructured  interview  group 
in  our  data.  In  addition  to  the  small  sample  size,  we  found  large  intra-group  differences  that  appear 
also  to  account  for  the  lack  of  statistically  significant  results.  Although  an  attempt  was  made  to 
match  the  groups  on  age,  gender,  and  education  level,  we  found  considerable  individual  differences 
in  the  students’  comfort  level  and  ease  in  conducting  interviews  (as  observed  by  the  investigators). 
This  resulted  in  large  standard  deviations  on  nearly  all  the  comparative  measures,  making  those 
findings  that  were  statistically  significant  difficult  to  interpret.  For  example,  in  rating  the  evaluation 
study  experience,  graduate  students  in  the  ACTA  group  for  both  the  firefighter  and  the  EW  study 
agreed  more  strongly  with  the  statement,  “I  felt  confident  in  my  ability  to  lead  an  interview”  (M  = 
4.18,  SD  =  .60)  than  the  graduate  students  who  conducted  unstructured  interviews  (M  =  3.25,  SD  = 
.87),  U  =  28.5,  p  =.02. 8.  In  the  firefighter  study,  the  ACTA  group  agreed  more  strongly  with  the 
statement,  “I  felt  I  had  sufficient  information  to  revise  the  course  materials”  ( M  =  4.00,  SD  =  .89) 
than  the  Unstructured  group  (M  =  2.67,  SD  =  1.03),  t  (10)  =  2.39,  p  =  .06.  These  statistical  analyses 
indicate  that  students  trained  to  use  ACTA  felt  more  confident  in  conducting  the  interviews  and 
were  more  confident  that  they  had  gathered  sufficient  information  to  revise  the  course  materials 
than  the  Unstructured  group.  However,  in  looking  at  the  large  standard  deviations,  it  becomes  clear 
that  some  students  in  each  group  were  confident,  whereas  others  were  not. 

In  other  cases,  the  means  indicate  very  little  difference  between  the  groups,  but  the  standard 
deviations  indicate  considerable  variance  within  the  groups.  For  example,  the  means  for  the  two 
groups  are  nearly  identical  in  response  to  the  question,  “Given  the  information,  I  found  the  revision 
of  course  materials  to  be  straightforward.”  However,  the  large  standard  deviations  indicate  that 
some  people  in  each  group  found  the  revision  of  the  course  materials  straightforward,  but  others  did 
not  (see  Table  5).  Given  the  small  sample  sizes  used  in  this  study,  it  is  clear  that  these  group 
difference  comparisons  are  not  very  robust.9 


8 A  Mann  Whitney  U,  which  is  free  of  variance  assumptions,  was  used  instead  of  a  t-test  because  there  was  not 
variance  in  the  responses  from  the  ACTA  students  in  the  EW  domain. 

9  An  additional  analyses  that  explored  group  differences  are  included  in  Appendix  C,  C.4. 
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Table  5 


Mean  Responses  to  Question  10  on  the  Interviewer  Questionnaire 


ACTA  Group 

Unstructured  Group 

Electronic 

Electronic 

Firefighting 

Warfare 

Firefighting 

Warfare 

Domain 

Domain 

Domain 

Domain 

Given  the  information,  I  found  the  revision  of  course 

3.67 

3.83 

3.00 

3.33 

materials  straightforward. 

(0.82) 

(1.47) 

(1.23) 

(1.03) 

Note.  Responses  refer  to  a  five  point  rating  scale,  where  1  =  strongly  disagree,  5  =  strongly  agree.  Standard  deviations 
are  in  parenthesis. 

One  potentially  confounding  factor  in  the  design  of  our  study  was  that  during  the  introductory 
workshop,  the  Unstructured  group  was  exposed  to  a  lecture  on  cognitive  elements.  Cognitive  Task 
Analysis,  and  how  to  fill  out  a  Cognitive  Demands  Table  before  conducting  interviews.  Some  of  the 
students  in  the  Unstructured  group  may  have  used  the  Cognitive  Demands  Table  to  structure  their 
interviews,  thus  reducing  the  gap  between  the  ACTA  group  and  the  Unstructured  group.  The 
implication  for  ACTA  is  that  the  Cognitive  Demands  Table  is  a  valuable  tool  for  framing  the  kinds 
of  information  that  the  interviewer  intends  to  elicit  from  the  SME. 

Although  we  considered  using  a  control  group  that  would  receive  no  introductory  workshop  on 
cognition,  we  found  this  to  be  impractical  given  that  we  wanted  to  compare  both  the  amount  of 
cognitive  information  elicited  in  the  interviews  and  the  quality  of  the  training  materials  produced.  In 
order  to  make  these  comparisons,  it  was  necessary  to  provide  training  in  how  to  create  a  Cognitive 
Demands  Table  and  how  to  produce  instructional  materials  to  all  of  the  participants  in  the  study. 

The  high  quality  ratings  (i.e.,  SME  ratings  of  importance  and  accuracy)  received  by  both 
interview  groups  indicate  that  an  exposure  to  the  concepts  underlying  Cognitive  Task  Analysis  and 
a  description  of  how  cognitive  task  analytic  data  can  be  applied  to  instruction  materials,  may  play  a 
large  role  in  learning  to  conduct  Cognitive  Task  Analyses.  Working  only  with  this  foundational 
material,  in  the  absence  of  exposure  to  actual  methodologies,  some  students  in  the  Unstructured 
group  were  able  to  gather  accurate,  relevant  cognitive  information  and  develop  useful  instructional 
materials. 

Qualitative  Analysis 

In  order  to  better  understand  the  variability  within  the  ACTA  group,  a  qualitative  analysis  was 
conducted  in  which  each  graduate  student’s  interview  notes  were  examined.  The  combined  output 
from  each  of  the  ACTA  interviews  was  examined  across  participants  by  two  researchers.  The  goal 
of  this  analysis  was  to  look  for  commonalities  and  discrepancies  across  the  data  set,  areas  of 
confusion,  and  indicators  that  participants  were  having  difficulty  conducting  the  interviews.  This 
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analysis  confirmed  the  variability  suggested  by  the  quantitative  data,  and  provided  us  a  better 
understanding  of  the  sources  of  variability. 

Our  understanding  of  where  participants  may  have  struggled  with  the  techniques  (thus 
increasing  the  variability  in  their  outputs)  led  to  a  revision  of  the  tools.  For  example,  we  found  that 
some  graduate  students  using  the  Task  Diagram  Interview  stmggled  to  get  the  SME  to  describe  the 
task  globally  in  terms  of  a  process,  and  would  sometimes  get  lost  in  details.  To  ameliorate  this 
problem,  we  reworded  the  initial  probe  from  “Please  decompose  this  task  into  subtasks....”  to 
“Think  about  what  you  do  when  you  (Task  of  Interest!  .  Can  you  break  this  down....”  In  addition, 
we  found  that  the  graduate  students  did  not  make  use  of  the  second  level  of  decomposition 
recommended  in  the  early  version  of  the  Task  Diagram  Interview,  so  we  changed  the  instructions  to 
say  one  should  go  to  second  level  of  decomposition  only  if  the  initial  Task  Diagram  is  too  broad. 

This  analysis  also  informed  us  that  students  were  having  trouble  changing  perspectives  in  the 
Knowledge  Audit  Interview.  The  early  versions  of  this  interview  instructed  the  interviewer  to  elicit 
an  example  (expert  perspective),  then  ask  why  that  activity  is  difficult  (novice  perspective),  and 
finally  ask  the  expert  to  describe  his/her  strategies  (expert  perspective).  To  reduce  the  confusion  we 
reordered  the  questions,  so  that  after  eliciting  an  example  (expert  perspective),  the  interviewer  asks 
the  expert  to  describe  his/her  strategies  (expert  perspective),  and  then  asks  why  the  task  would  be 
difficult  for  an  inexperienced  person  (novice  perspective).  The  new  order  requires  only  one  switch 
in  perspective. 

Analysis  of  the  Simulation  Interview  notes  indicated  that  students  were  having  difficulty 
following  the  SME’s  narrative.  This  led  us  to  simplify  the  Simulation  Interview  procedure.  Early 
versions  of  this  method  included  a  probe  regarding  alternative  courses  of  action/situation 
assessments  and  a  probe  about  potential  errors  an  inexperienced  person  might  make.  Graduate 
students  had  difficulty  discriminating  the  types  of  information  these  two  probes  might  elicit  and 
thus  ended  up  with  considerable  overlap.  We  eliminated  the  alternative  courses  of  action/situation 
assessments  probe  to  reduce  redundancy  and  simplify  the  method.  In  addition,  we  rearranged  the 
probes  so  that  it  would  be  easier  for  the  SME  to  tell  his/her  story.  Initially,  the  first  probe  after 
identifying  the  major  decision  points  focused  on  situation  assessment  followed  by  a  probe 
addressing  courses  of  action.  Although  one  typically  assesses  a  situation  before  determining  a 
course  of  action,  in  telling  a  story  people  tend  to  describe  actions  first  and  then  fill  in  details  about 
what  led  to  that  action.  The  order  of  the  first  two  probes  was  reversed  to  accommodate  the 
interviewees’  story-telling  tendencies. 

Discussion 

The  findings  presented  here  indicate  that,  after  a  2-hour  workshop  defining  Cognitive  Task 
Analysis  and  a  6-hour  workshop  introducing  the  ACTA  techniques,  graduate  students  were  able  to 
conduct  interviews  with  SMEs  and  elicit  important,  accurate  cognitive  information  that  was  easily 
translated  into  instructional  materials.  Furthermore,  subjective  reports  from  the  graduate  students 
indicate  that  the  techniques  are  easy  to  use,  flexible,  and  provide  clear  output.  Our  belief  is  that 
professional  Instructional  Designers  and  system  designers  will  do  even  better  than  our  graduate 


39 


students,  given  that  they  will  have  more  concrete  goals  for  use  of  the  cognitive  information  and 
more  experience  generating  applications. 

Although  an  attempt  has  been  made  here  to  establish  the  reliability  and  validity  of  the  ACTA 
methods,  we  are  aware  that  no  well-established  metrics  exist.  The  need  to  test  Cognitive  Task 
Analysis  methods  in  real-world  settings  with  real-world  tasks  greatly  reduces  the  level  of  control 
one  has  over  the  many  sources  of  variability.  Factors  that  are  difficult  to  control  include  the  fact  that 
some  people  seem  to  be  more  predisposed  to  be  good  interviewers  than  others.  In  addition,  some 
SMEs  are  more  articulate  and  easier  to  focus  than  others.  Given  the  variability  among  humans  in 
both  the  interviewer  and  the  SME  roles,  it  will  be  important  to  answer  such  questions  as:  Does  an 
SME  report  the  same  examples  and  the  same  details  when  asked  the  same  question  later  in  time?  Do 
the  Cognitive  Task  Analysis  techniques  elicit  the  same  types  of  information  when  used  by  different 
interviewers?  Do  independent  practitioners  generate  the  same  materials  based  on  Cognitive  Task 
Analysis  interviews?  The  results  of  this  evaluation  study  strongly  suggest  that  the  ACTA  tools  are 
reliable  and  valid,  but  further  research  is  needed  to  establish  meaningful  metrics  to  assess  the 
reliability  and  validity  of  Cognitive  Task  Analysis  tools  with  experienced  professionals. 

Another  point  to  consider  is  that  although  the  ACTA  methods  have  been  shown  to  elicit 
important,  accurate,  cognitive  information,  we  have  yet  to  assess  what  is  lost  using  these 
streamlined  techniques.  It  is  our  belief  that  a  tradeoff  exists:  the  more  streamlined  and 
proceduralized  Cognitive  Task  Analysis  techniques  become,  the  less  powerful  they  are.  Our 
suspicion  is  that  the  ACTA  techniques  gather  less  comprehensive  information  than  more  systematic 
techniques  such  as  Hall  et  al.’s  PARI  method  (1994)  and  Gordon  and  Gill’s  (1992)  conceptual 
graph  analysis,  and  that  the  information  gathered  is  more  superficial  than  that  gathered  using  the 
Critical  Decision  method  (Klein,  Calderwood,  &  MacGregor,  1989)  or  Rasmussen’s  (1986) 
cognitive  analysis.  In  spite  of  the  limitations  of  streamlined  Cognitive  Task  Analysis  procedures, 
the  ACTA  techniques  provided  graduate  students  sufficient  tools  to  identify  key  cognitive  elements 
and  develop  useful  training  materials.  Until  better  metrics  exist,  however,  it  will  be  difficult  to 
objectively  assess  what  is  lost  and  what  is  gained  via  different  techniques. 

It  is  also  important  to  point  out  the  potential  impact  of  ignoring  or  minimizing  cognitive  issues 
in  complex  tasks  requiring  high  degrees  of  cognitive  skill.  Despite  the  promise  of  automated  and 
intelligent  systems,  the  human  decision  maker  will  always  play  a  role  in  systems  where  uncertainty 
and  ambiguity  exist.  The  consequences  of  not  training  operators  to  acquire  the  cognitive  skills 
required,  or  not  designing  systems  to  support  human  problem  solving  and  decision  making  can  be 
dire,  as  illustrated  by  disasters  such  as  Three  Mile  Island,  the  USS  Vincennes,  and  Kings  Cross 
Station,  to  name  but  a  few  (Reason,  1990).  Our  results  indicate  that  with  minimal  training, 
relatively  inexperienced  interviewers  can  be  taught  how  to  use  the  ACTA  tools  to  obtain  useful  and 
important  cognitive  information  in  domains  with  which  they  are  unfamiliar.  This  information  will 
be  useful  in  designing  training  and  building  systems  to  minimize  errors. 

The  ACTA  methodology  was  originally  taught  in  a  workshop  format,  which  allowed  the 
workshop  presenters  to  tailor  the  methods  to  the  audience  and  add  personal  anecdotes  to  the 
instruction.  This  also  meant  that  no  two  ACTA  workshops  were  the  same.  This  made  research  into 
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the  reliability  of  the  methods  even  more  difficult  to  evaluate.  However,  more  recently  the  ACTA 
training  materials  have  been  produced  on  a  compact  disk-based,  multimedia  training  tool  (Militello, 
Hutton,  &  Miller,  1996).  This  tool  provides  the  learner  with  the  reasons  for  undertaking  a  Cognitive 
Task  Analysis,  an  introduction  to  cognition  and  expertise,  a  tutorial  on  the  three  ACTA  techniques, 
an  explanation  of  the  Cognitive  Demands  Table,  and  sample  applications.  See  the  Stand-Alone 
Instruction  for  a  detailed  description  of  the  software. 


Stand-Alone  Instruction 

This  section  describes  the  stand-alone  instructional  package  intended  to  train  Instructional 
Systems  Specialists  (ISSs)  and  other  Instructional  Designers  in  how  to  use  the  Applied  Cognitive 
Task  Analysis  (ACTA)  techniques  and  apply  the  information  obtained  via  ACTA  to  training.  In 
addition,  this  section  includes  a  detailed  discussion  of  the  design  rationale  for  the  ACTA 
Instructional  Software. 

The  ACTA  instructional  package  includes  the  ACTA  Instructional  Software  housed  on  a  CD- 
ROM,  a  pamphlet  outlining  the  content  of  the  software,  and  three  job  aids  corresponding  to  each  of 
the  ACTA  tools.  The  pamphlet  and  job  aids  are  included  in  Appendix  D.  The  ACTA  Instmctional 
Software  is  described  below. 

Software  Description 

The  ACTA  Instmctional  Software  contains  thorough  audio  and  written  descriptions  of  the  three 
ACTA  interview  techniques  along  with  visual  displays  that  allows  the  participant  to  learn  how  to 
conduct  interviews  with  Subject  Matter  Experts  (SMEs).  The  ACTA  Instmctional  Software  is  made 
up  of  three  main  sections.  They  are  the  Introduction,  Interview  Methods,  and  Applications  sections. 

The  Introduction  gives  a  brief  overview  of  what  is  contained  in  the  software.  First,  ACTA  is 
defined  and  the  essentials  of  the  method  are  explained.  Included  in  the  essentials  are: 

•  What  ACTA  helps  you  do. 

•  Understanding  cognition  and  examples  of  cognitive  elements. 

•  Novice  vs.  expert  performance  differences. 

•  The  limitations  of  behavioral  task  analysis  when  compared  with  ACTA. 

Secondly,  the  Introduction  section  presents  an  overview  of  the  Task  Diagram,  the  Knowledge 
Audit,  and  the  Simulation  Interview.  And  lastly,  the  Introduction  concludes  with  a  several  tips  on 
how  to  get  started  on  conducting  ACTA  interviews.  These  include: 

•  Knowledge  of  the  domain. 

•  Possible  sources  for  obtaining  this  knowledge. 
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•  Finding  SMEs,  and  how  many  SMEs  to  interview. 

•  Interview  supplies. 

•  Tape  recording  tips. 

•  Planning  a  timeline. 

The  Interview  Methods  section  describes  the  three  ACTA  interview  techniques  (the  Task 
Diagram,  the  Knowledge  Audit,  and  the  Simulation  Interview),  and  also  Tricks  of  the  Trade  for 
each.  Each  interview  method  is  defined  and  describes  steps  on  how  to  get  started.  An  example  of 
each  interview  method  and  tips  for  conducting  these  methods  are  described.  Also  included  for  each 
interview  method  is  a  practice  session  where  the  participant  listens  to  a  segment  of  a  mock 
interview  and  then  determines  what  the  interviewer  did  right  or  wrong.  Lastly,  the  software  presents 
Tricks  of  the  Trade,  which  pose  things  to  remember  when  conducting  the  interviews.  They  include: 
the  use  of  diagrams  and  drawings,  method  strengths,  persistence,  teaming,  time  issues,  preparation, 
how  to  deal  with  conflicting  information,  controlling  the  interview,  and  a  reminder  to  be  flexible 
when  using  the  ACTA  techniques. 

The  Applications  section  includes  a  discussion  and  examples  of  the  following: 

•  Advance  Organizer. 

•  Cognitive  Demands  Table. 

•  Learning  Objectives. 

•  Simulations  and  Scenarios. 

The  Advance  Organizer  is  an  instructional  tool  that  serves  as  a  bridge  between  known  material 
and  new  material.  Information  gathered  using  the  Task  Diagram  interview  can  feed  directly  into  an 
Advance  Organizer.  Elements  of  an  effective  Advance  Organizer  are  described.  Steps  to  developing 
the  Cognitive  Demands  Table  as  a  tool  for  analyzing  and  consolidating  data  from  interviews  are 
included.  Examples  of  cognitive  learning  objectives  derived  from  a  Cognitive  Demands  Table  are 
presented.  Lastly,  in  Simulations  and  Scenarios,  guidelines  and  techniques  are  given  for  creating  an 
instructor  guide  that  emphasizes  the  use  of  simulations  and  scenarios  to  develop  cognitive  skills. 

Design  Rationale  for  the  ACTA  Instructional  Software 

The  purpose  of  this  section  is  to  describe  the  goals  and  rationale  for  the  design  of  the  stand¬ 
alone  instructional  package.  The  section  will  describe  the  challenge  of  designing  this  product,  the 
design  goals,  and  the  issues  that  impacted  the  final  design.  It  will  also  describe  the  design  process 
that  occurred  between  Klein  Associates  and  the  instructional  software  developer. 


42 


Goals  of  the  Tool 


The  goal  of  this  project  was  to  develop  a  stand-alone  instructional  tool  that  would  allow  the 
learner  to  do  four  things: 

•  Understand  the  underlying  concepts  of  cognition  and  expertise  that  drive  the  ACTA  tools. 

•  Conduct  Applied  Cognitive  Task  Analysis  (ACTA)  interviews. 

•  Consolidate  and  represent  the  data  from  those  interviews. 

•  Apply  the  data  to  training. 

The  intended  audience  is  the  Navy  ISS  community.  It  was  anticipated  that  stand-alone 
instruction  would  be  used  on  an  individual  basis  as  opposed  to  being  used  by  several  people  at  the 
same  time.  This  meant  that  the  interaction  would  be  between  a  single  learner  and  the  instructional 
package  as  opposed  to  allowing  for  interaction  between  study  partners  to  practice  interview 
techniques. 

Challenges 

There  were  several  challenges  posed  by  this  effort.  The  first  challenge  was  to  expose  the  learner 
to  key  elements  from  the  cognition  and  expertise  literature  that  would  help  in  implementing  the 
ACTA  techniques.  Our  experiences  in  the  ACTA  workshops  indicated  that  people  need  an 
understanding  of  the  types  of  information  they  are  searching  for  if  they  are  to  use  the  ACTA 
techniques  successfully.  A  further  challenge  was  to  teach  the  ACTA  techniques,  which  included  not 
only  the  knowledge  elicitation  interviews  but  also  the  consolidation  and  integration  of  this 
information  into  some  sort  of  representation.  Although  teaching  the  learner  basic  interviewing  skills 
was  not  a  primary  objective  of  the  instructional  software,  it  was  important  to  provide  some  guidance 
regarding  how  to  conduct  a  knowledge  elicitation  interview.  A  final  challenge  was  to  discuss  and 
provide  examples  of  how  data  gathered  using  ACTA  could  be  used  to  provide  improved  training 
techniques  that  incorporate  the  cognitive  skills  and  demands  highlighted  by  the  knowledge 
elicitation  interviews. 

Cognition  and  Expertise.  One  important  lesson  learned  via  the  many  workshops  conducted  is 
that  people  must  have  in  mind  what  types  of  information  they  are  looking  for  if  they  are  to  be 
successful  in  using  the  ACTA  techniques.  It  was  therefore  critical  that  we  identify  key  elements 
from  the  cognition  and  expertise  literature,  and  provide  examples  of  how  those  elements  might  play 
out  in  several  domains.  Although  the  ACTA  techniques  provide  questions  that  begin  to  unlock  the 
doors  to  expertise,  the  interviewer  must  recognize  the  cognitive  elements  and  be  able  to  make  sense 
of  them.  Examples  of  cognitive  elements  in  several  domains  are  provided  in  the  ACTA  Instructional 
package  so  that  the  learner  will  have  a  better  understanding  of  what  types  of  information  s/he  is 
looking  for. 


43 


In  large  part,  the  flow  of  the  ACTA  interviews  is  dictated  by  the  interviewee  and  the 
responses  s/he  provides.  This  means  that  questioning  is  dependent  on  the  previous  response  and  no 
set  of  rigid  rules  or  procedures  will  allow  the  interviewer  to  conduct  these  interviews  successfully. 
The  interviewer  can  rarely  ask  one  question,  which  leads  to  a  single  response  that  can  then  be 
recorded  allowing  the  interviewer  to  move  on  to  the  next  question.  The  interviewer  has  to  interpret 
the  response  based  on  an  understanding  of  expert  performance,  and  assess  whether  the  response 
includes  relevant  cognitive  information.  Often  the  initial  response  will  need  clarification  or  more 
detail.  The  ACTA  techniques  require  the  interviewer  to  deepen  beyond  the  initial  response  in  most 
cases.  The  interviewer  provides  direction  for  the  interview,  including  which  portions  of  the  task  to 
deepen  and  how  deeply  to  pursue  each  element.  These  judgments  are  based  on  an  understanding  of 
the  SME’s  task  and  on  an  understanding  of  the  clues  that  indicate  expert  performance  and  expert 
decision  making.  The  interviewer  must  know  what  type  of  information  s/he  is  looking  for  and 
whether  the  SME’s  response  satisfies  that  requirement. 

The  ACTA  Techniques.  In  the  ACTA  workshops  each  of  the  ACTA  techniques  was 
explained,  followed  by  a  demonstration  of  the  technique,  and  an  opportunity  for  the  participants  to 
practice  the  technique.  Our  intention  was  to  follow  a  similar  format  in  the  stand-alone  instructional 
package.  Considerable  time  was  spent  investigating  different  potential  media  to  understand  the 
strengths  and  weaknesses  of  each,  and  to  determine  which  would  allow  us  to  provide  adequate 
explanations,  demonstrations,  and  practice  opportunities.  Although  the  instructional  content  for 
teaching  the  ACTA  technique  had  been  well-rehearsed  and  refined,  the  transition  to  stand-alone 
instruction  required  a  repackaging  and  condensing  of  the  information. 

Our  experience  teaching  the  ACTA  techniques  in  workshops  highlighted  the  fact  that,  in  order 
for  people  to  successfully  use  the  ACTA  techniques,  they  must  have  basic  cognitive  interviewing 
skills.  Teaching  the  subtle  skills  of  interviewing  through  any  medium  is  difficult.  An  interview 
requires  many  interpersonal  interaction  skills  as  well  as  skills  to  elicit  the  required  information.  The 
ACTA  interviews  are  not  only  demanding  from  the  perspective  of  having  to  understand  the 
individual’s  area  of  expertise,  but  also  from  the  perspective  of  being  able  to  get  below  a  surface 
level  of  procedures  and  rules — to  the  level  of  understanding  what  it  is  that  makes  the  individual  an 
expert  in  that  area  of  expertise. 

There  are  many  aspects  of  conducting  an  interview  that  require  skill  and  confidence.  For 
example,  the  interviewer  must  first  establish  rapport  with  the  interviewee  so  that  the  interviewee 
feels  comfortable  talking  about  his/her  job  and  helping  the  interviewer  understand  the  job. 
Depending  on  the  task  being  investigated,  the  interview  may  need  to  cover  a  large  subject  area  in  a 
limited  amount  of  time.  This  requires  interview  time  management  skills.  The  interviewer  must  be 
able  to  listen  for  and  understand  the  content,  and  at  the  same  time  listen  for  clues  that  will  lead  to 
the  next  question.  While  doing  this  the  interviewer  must  try  to  document  and  record  as  much  as 
possible  from  the  interview. 

Data  Analysis  and  Consolidation.  After  conducting  the  ACTA  techniques  with  a  number  of 
SMEs,  the  Instructional  Designer  must  find  a  way  to  analyze  the  data,  extracting  that  which  will 
inform  the  development  of  a  course  and/or  training  materials,  and  consolidating  it  into  a  usable 
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format.  We  recommend  the  use  of  a  Cognitive  Demands  Table  to  frame  this  process.  This  process 
requires  the  Instructional  Designer  to  have  firmly  in  mind  the  type  of  course  and/or  materials  s/he 
intends  to  build.  In  addition,  the  Instructional  Designer  must  be  able  to  recognize  where  the  key 
cognitive  elements  of  this  task  are,  based  upon  what  s/he  has  learned  via  the  ACTA  interviews.  This 
process  of  data  analysis  and  consolidation  calls  on  the  Instructional  Designer’s  understanding  of 
cognition  and  expertise,  and  his/her  skills  as  an  Instructional  Designer. 

Applying  the  Data  to  Training.  The  final  step  is  to  actually  design  the  course  or  create  the 
training  materials.  In  the  past,  many  of  the  training  techniques  used  by  the  Navy  have  emphasized 
behavioral  skills.  It  is  critical  in  the  environment  in  which  the  ISS  works  to  be  able  to  measure  and 
evaluate  each  skill  that  is  taught.  Feedback  we  received  from  participants  in  our  workshops 
indicated  that  a  discussion  of  how  to  apply  data  gathered  using  ACTA  to  training  would  be  needed 
for  acceptance  from  our  user  community. 

Solutions 

Our  first  major  decision  was  to  decide  on  an  instructional  medium.  We  had  found  no  analogues 
to  teaching  interviewing  techniques.  The  closest  candidates  were  customer  service  videos,  which 
used  a  narrator  “customer  service  expert”  interspersed  with  short  scenarios  to  illustrate  points. 

This  led  us  to  consider  video  instruction.  Initially,  this  format  was  appealing  because  it  would 
provide  us  the  ability  to  demonstrate  many  of  the  interpersonal  skills  and  the  interviewing  skills 
required  for  ACTA.  It  could  be  engaging,  and  would  allow  the  use  of  text  within  the  video  to 
highlight  key  points  and  present  more  abstract  ideas.  However,  further  investigation  into  video 
technology  and  our  increasing  experience  in  teaching  the  ACTA  techniques  via  workshops 
highlighted  the  limitations  of  video  technology.  Video  is  a  very  linear  medium  with  little  flexibility 
in  terms  of  how  the  information  is  presented.  It  also  does  not  give  the  learner  control  over  the  pace 
or  presentation  of  the  information.  Video  could  be  engaging  for  short  periods,  but  allows  for  little 
variety  in  presentation  techniques,  so  that  over  time  it  becomes  less  engaging.  This  technique 
worked  well  in  the  customer  service  videos  we  were  using  as  an  analog,  but  we  did  not  believe  it 
could  accommodate  the  increased  time  requirements  of  our  instructional  package  without  becoming 
monotonous,  and  it  did  not  provide  the  flexibility  necessary  to  teach  the  ACTA  techniques. 

The  next  option  we  considered  was  the  use  of  multimedia  software.  Computer-based  technology 
allows  the  integration  of  computer  text  and  graphics,  with  video  and  sound,  as  well  as  providing  the 
learner  with  a  large  amount  of  control  and  flexibility  in  interacting  with  the  material.  Multimedia 
software  would  provide  the  ability  to  use  video  segments  combined  with  the  flexibility  of  non¬ 
linear,  self-paced  learning.  This  would  mean  that  the  learner  would  be  able  to  choose  his/her  own 
path  through  the  learning  environment.  Learners  would  be  able  to  review  portions  of  the  instruction 
easily  and  flexibly,  and  would  be  able  to  quit  and  rejoin  the  instruction  where  they  left  off  or  at  any 
other  point  in  the  program.  It  would  allow  for  learner  interaction  with  the  material.  This  format  also 
provided  an  added  bonus  of  being  very  portable,  and  thus  easily  accessible  to  potential  users 
throughout  the  Navy. 
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Our  initial  enthusiasm  for  this  technology  was  soon  tempered  by  the  reality  of  technology 
limitations.  The  use  of  video  requires  a  large  amount  of  memory  to  implement,  and  without  state- 
of-the-art  equipment  video  quality  quickly  degrades.  This  meant  that  in  order  to  illustrate  the 
interview  techniques  using  video,  we  could  easily  consume  most  of  the  memory  on  a  single  CD- 
ROM  and  still  end  up  with  low  resolution,  grainy  images  that  would  be  difficult  to  interpret. 
Elements  such  as  facial  expressions  and  other  nonverbal  indicators  of  interpersonal  skills  related  to 
interviewing  would  be  lost.  In  addition,  the  programmers  indicted  that  without  state-of-the-art 
equipment  to  play  the  CD,  sound-video  synchronization  would  be  degraded.  Memory  constraints 
would  reduce  the  length  of  each  video  clip  to  between  20  and  30  seconds.  This  small  amount  of 
time  would  reduce  our  ability  to  convey  the  subtleties  and  difficulties  of  conducting  these 
interviews. 

Although  technology  limitations  ruled  out  the  use  of  video  within  the  instructional  software,  the 
flexibility  of  the  medium  still  qualified  multimedia  software  as  the  most  effective  format  for 
teaching  the  ACTA  techniques.  To  replace  the  use  of  video,  instructional  programmers 
recommended  the  use  of  still  photos  accompanied  by  audio  interview  segments.  The  use  of  these  is 
described  below. 

The  Software  Design 

In  establishing  the  look  and  feel,  architecture  and  functionality  of  the  system,  we  worked  very 
closely  with  instructional  software  designers  from  Multimedia  Learning,  Inc.  Klein  Associates 
provided  the  content  and  architectural  framework  for  the  system.  The  instructional  developers 
provided  much  of  the  creative  input  and  offered  options  for  functionality.  In  designing  the  software, 
instructional  principles  as  well  as  human  factors  design  principles  were  taken  into  consideration. 

Our  primary  design  goals  were  for  the  instructional  package  to  be: 

•  Appealing  to  the  learner. 

•  Easy  to  navigate  through. 

•  Consistent  in  its  interactions  and  prompts. 

•  Logical  in  the  progression  of  ideas,  methods,  and  applications. 

•  Easy  to  understand,  despite  subtle  concepts. 

•  Engaging. 

Appeal  to  the  Learner 

The  instructional  package  had  to  generate  immediate  interest  from  the  learner.  The  first  aspect 
of  this  is  the  initial  look  and  feel  of  the  system — its  visual  appeal.  We  tried  several  different 
graphical  looks  for  the  package,  with  different  styles  of  graphics  and  different  color  schemes.  The 
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goal  was  to  produce  a  look  that  was  engaging.  The  graphics  had  to  be  visually  interesting,  but  not 
distracting. 

Another  critical  aspect  in  appealing  to  the  learner  was  to  identify  a  need,  and  to  demonstrate 
how  ACTA  could  fill  that  need.  The  need  in  this  case  is  to  be  able  to  extract  information  from 
experts  about  the  cognitive  skills  needed  to  conduct  a  specific  task,  capture  that  expertise  and 
cognitive  skill,  and  then  apply  that  information  in  the  form  of  training.  The  first  several  frames  and 
narrations  of  the  package  introduce  the  need  and  the  benefits  of  ACTA  in  addressing  that  need. 

Easy  to  Navigate 

The  second  key  element  in  the  design  of  the  system  was  the  ease  of  navigation  throughout  the 
system.  This  would  support  the  usability  of  the  system  and  allow  for  flexible  use  by  learners. 
Critical  to  implementing  a  system  that  would  be  easy  to  navigate  is  designing  a  straightforward 
architecture.  The  software  was  divided  into  three  primary  modules:  an  introduction,  the  ACTA 
techniques,  and  the  analysis  and  application  of  the  data  to  training.  Each  of  these  three  primary 
modules  consisted  of  three  to  four  lessons.  This  simple  structure  means  that  it  is  easy  for  the  learner 
to  navigate  within  the  software.  (See  Figure  10.) 

Navigation  is  aided  not  only  by  the  shallow  hierarchy,  but  also  by  the  constant  presence  of  a 
“side-bar”  with  several  action  buttons.  The  first  is  a  “Quit”  button,  which  takes  you  directly  out  of 
the  program.  A  “Menu”  button  takes  you  directly  back  to  the  main  menu  screen.  A  “Forward” 
button  allows  you  to  skip  a  frame  within  a  lesson.  A  “Back”  button  allows  you  to  move  back  a 
frame  within  a  lesson,  and  a  “Repeat”  button  allows  you  to  repeat  the  previous  frame. 

In  addition  to  these  navigation  aids,  there  is  also  a  “Help”  button,  which  appears  at  the  main 
menu  screen.  This  help  button  introduces  the  basics  of  interacting  with  this  type  of  interface  and 
describes  the  function  of  each  button  on  the  “side-bar.” 
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Figure  10.  ACTA  software  hierarchy. 


Logical  Progression 

Navigation  and  usability  are  also  improved  by  the  logical  flow  and  progression  of  lessons  within 
the  instructional  package.  The  non-linear  aspect  of  this  form  of  computer-based  training  allows  the 
learner  to  choose  his  or  her  own  path  through  the  modules  and  lessons.  However,  the  layout  of  the 
module  and  lesson  structure  was  designed  to  provide  a  logical  progression  through  the  introduction 
of  concepts,  the  interview  methods,  and  applications  of  the  data  to  training.  The  learner  will  gain  the 
most  from  ACTA  by  following  this  logical  progression  through  the  concepts,  methods,  and 
applications  the  first  time  through. 
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The  heart  of  the  package  is  the  middle  module,  which  presents  the  three  interview  tools  and 
provides  opportunity  to  see  the  interview  in  action  through  photo-snapshot  demonstrations.  The 
flexible  format  of  ACTA  will  allow  a  learner  to  go  straight  to  whichever  section  s/he  is  most 
interested  in  revisiting. 

Easy  to  Understand 

A  further  goal  of  the  software  package  was  that  it  would  be  usable  by  ISSs  and  private  sector 
Instructional  Designers,  who  may  have  varying  educational  backgrounds  and  work  in  widely 
diverse  domains.  This  meant  that  the  flow  had  to  be  logical  as  explained  above,  and  that  the 
language  and  key  concepts  had  to  be  defined  up  front.  As  has  been  explained  earlier,  an 
understanding  of  these  key  concepts  and  an  integration  of  those  key  concepts  into  the  interviewer’s 
knowledge-base  is  important  in  conducting  these  types  of  interviews.  Through  the  multiple 
workshops  and  evaluations  we  learned  which  terms  and  concepts  were  the  most  troublesome  for 
people  to  learn.  Our  user  analysis  helped  to  identify  terms  that  would  be  unfamiliar  to  our  user 
population.  As  far  as  possible,  we  attempted  to  use  words  used  in  everyday  language  or  words  that 
could  be  understood  within  the  context  of  the  usage. 

Engaging 

One  big  advantage  of  this  form  of  instruction  is  that  the  multimedia  format  allows  the  use  of 
animation,  color,  sound,  graphics,  etc.  to  create  an  engaging  learning  environment.  This  multimedia 
format  also  allows  for  interactions  with  the  computer,  which  increase  learner  engagement.  Our 
intention  was  to  take  advantage  of  the  capability  of  multimedia  technology  to  provide  an  interesting 
visual  environment  with  meaningful  graphics  and  text  to  represent  the  concepts,  to  provide  an 
interesting  audio  account  of  the  material,  and  to  allow  the  learner  to  control  and  interact  with  the 
program.  The  goal  was  to  avoid  recreating  a  video-on-a-PC.  In  other  words,  to  avoid  having  the 
learner  sitting  in  front  of  a  PC  watching  the  instructional  package  play  itself  out  much  like  sitting  in 
front  of  a  video.  The  next  section  describes  our  instructional  approach  for  the  instructional  package, 
followed  by  the  focal  interactions  that  were  intended  to  engage  and  instruct  the  learner  in  Cognitive 
Task  Analysis. 

Instructional  Approach 

The  basic  instructional  approach  taken  is  one  of  encouraging  the  learner  to  model  the  behavior 
of  an  expert  interviewer.  A  situation  is  presented,  example  interview  questions  and  probes  are  asked 
by  a  model  interviewer,  the  interviewee  responds,  and  the  interviewer  records  model  notes  of  the 
responses  into  the  appropriate  table  format.  The  learner  is  encouraged  to  watch  the  interaction  and 
model  the  interviewer’s  behavior.  After  the  appropriate  interviewer  behavior  is  modeled,  exercises 
are  provided  in  which  the  learner  is  asked  to  type  in  an  answer  or  choose  the  correct  response  from  a 
list.  The  learner’s  responses  are  then  compared  with  model  responses  supplied  by  the  software. 

In  the  Introduction  the  learner  is  led  though  the  instruction  by  a  narrator.  Contemporary,  stylized 
graphics  are  used  to  represent  the  key  concepts  and  how  they  are  related  along  with  text  to  further 
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explain  the  concepts.  The  audio  text  and  written  text  match  closely  but  not  exactly,  thus  providing 
two  different  presentations  of  the  material. 

In  the  Interviewing  Methods  section  the  narrator  presents  the  basic  procedure,  probes,  and  data 
collection  tools  for  conducting  each  interview.  Each  interview  technique  is  illustrated  with  an 
example  using  an  interviewer  or  Instructional  Designer  who  is  interviewing  a  Fireground 
Commander  about  firefighting.  The  interview  process  used  in  the  example  is  mapped  out  for  the 
learner  as  it  progresses  through  each  of  the  steps. 

In  the  Interviewing  in  Action  section,  the  learner  is  introduced  to  Lynn  Thomas,  a  Navy  ISS, 
and  to  Joe  Harris,  a  Navy  EW  Technician.  An  example  of  each  interview  method  is  provided  with 
Lynn  interviewing  EW1  Harris  about  EW  operations.  The  interview  setting  is  presented  by  multiple 
photograph  snapshots  of  the  interview  situation,  with  Lynn  sitting  at  the  desk  or  standing  at  a 
whiteboard  recording  the  responses,  and  EW  Harris  sitting  at  the  desk  answering  Lynn’s  questions. 
Beneath  the  photographic  representation  of  the  interview,  the  table  in  which  Lynn  is  recording  EW 
Harris’s  responses  is  presented.  Examples  of  the  notes  that  Lynn  is  taking  appear  in  the  tables. 

Following  each  clean  run  through  an  interview,  the  learner  is  then  presented  with  a  number  of 
tough  interview  scenarios  with  tips  on  how  to  handle  these  situations.  Some  of  these  interview  tips 
include  exercises  where  the  learner  is  required  to  type  in  his  or  her  response  to  a  question  and  then 
compare  it  to  the  correct  answer. 

The  final  section.  Applications,  reverts  back  to  the  narrator  and  the  stylized  graphics  to  present 
how  to  consolidate  the  information  from  multiple  interviews,  and  how  to  convert  that  information 
into  training  materials. 

Interactions 

There  are  four  primary  interactions  in  this  program:  interactions  for  navigation;  interactions 
where  the  learner  must  click  on  a  graphical  object  to  get  more  information  about  that  object; 
interactions  where  the  learner  must  type  a  response  to  a  question;  and  interactions  where  the  learner 
must  click  on  one  or  more  choices  in  response  to  a  question. 

The  purpose  of  these  interactions  is  to  allow  the  learner  to  progress  through  the  learning 
environment  and  pick  up  the  key  concepts  while  remaining  engaged  with  the  presentation. 

Navigation 

The  learner  can  click  on  any  of  the  “sidebar”  buttons  to  go  forward,  skip  through  a  frame,  skip 
back  a  frame,  or  repeat  a  frame.  The  learner  can  also  return  directly  to  the  main  menu  by  clicking  on 
the  “Menu”  button.  These  interactions  are  the  basis  of  progressing  through  the  instruction. 
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Get  More  Information 


The  learner  is  sometimes  given  the  option  of  clicking  on  an  object  on  the  screen  (a  tape 
recorder,  a  photograph,  the  simulation  text)  in  order  to  obtain  more  information.  This  additional 
information  is  not  presented  in  the  body  of  the  instruction  as  it  is  not  critical  to  learning  the  skills  of 
Cognitive  Task  Analysis. 

Type  Response  to  a  Question 

During  the  Interviewing  in  Action  segment  of  the  instruction,  the  learner  is  provided  with  an 
interview  scenario  played  out  by  an  interviewer  and  interviewee,  and  the  learner  is  asked  to  critique 
the  interviewer’s  performance.  The  learner  is  asked  to  type  in  what  the  interviewer  did  well  or  did 
badly  in  handling  the  challenging  interview  situation.  Once  the  learner  has  responded,  the  learner 
can  click  on  an  “ideal”  response  in  order  to  compare  their  own  answer  to  the  model  response. 

Multiple  Choice 

Following  the  presentation  of  a  concept  or  concepts  during  the  instruction,  the  learner  is 
presented  with  a  statement  about  the  concept  and  a  list  of  sentences,  which  are  either  related  to  the 
initial  concept  presented  or  are  unrelated  to  the  concept  presented.  The  learner  is  asked  to  click  on 
all  the  statements  that  are  true  about  the  initial  statement.  When  the  learner  is  finished,  feedback  is 
provided.  After  two  attempts,  if  the  learner  has  not  identified  the  correct  statements,  the  correct 
answers  are  provided. 

ACTA  Software  Review 

The  final  phase  of  this  project  called  for  a  review  of  the  ACTA  Instructional  Software. 
Instructional  design  professionals  were  given  the  opportunity  to  review  the  ACTA  Instructional 
Software.  The  purpose  of  the  software  review  was  for  the  instructional  design  professionals  to 
evaluate  the  software  to  determine  whether  or  not  professionals  in  the  instructional  design  field 
could  learn  how  to  use  and  implement  the  ACTA  techniques  from  this  stand-alone  instructional 
material.  For  the  software  review,  participants  were  instructed  to  review  the  instructional  software 
to  become  familiar  with  three  interview  techniques  (i.e..  Task  Diagram,  Knowledge  Audit,  and 
Simulation  Interview)  and  one  technique  for  summarizing  interview  data  (i.e.,  the  Cognitive 
Demands  Table),  and  to  fill  out  a  Software  Review  Questionnaire.10 


After  reviewing  the  software,  one  participant  then  utilized  the  techniques  by  conducting  interviews  with  two 
SMEs  and  generating  learning  objectives  for  a  potential  course.  The  participant’s  subjective  evaluation  of  the  usability 
of  the  ACTA  techniques  was  positive.  Of  the  learning  objectives  that  were  developed  based  on  her  interviews,  both 
SMEs  rated  the  learning  objectives  as  accurate  and  containing  important  information  for  learners. 
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Method 


Participants.  Names  of  potential  participants  were  provided  through  the  Navy  COTR,  Dr. 
Josephine  Randel,  and  through  various  contacts  developed  during  the  project.  From  the  names 
provided,  Klein  Associates  researchers  contacted  several  instructional  design  professionals  to  ask  if 
they  would  be  interested  in  conducting  a  software  review  of  the  ACTA  Instructional  Software.  A 
total  of  twelve  participants  (7  males  and  5  females)  evaluated  the  ACTA  Instructional  Software. 
Three  of  the  participants  were  Navy  ISSs  (3  male)  and  six  of  the  participants  (3  males  and  3 
females)  were  commercial  Instructional  Designers.  Three  participants  (2  male  and  1  female)  who 
were  not  Instructional  Designers  but  were  recommended  because  of  their  experience  as  instructors 
who  provide  input  to  courses  in  their  own  areas  of  expertise  were  also  asked  to  review  the  ACTA 
Instructional  Software. 

Materials.  Materials  for  the  software  review  included  a  cover  letter  and  instructions,  which  took 
each  participant  step  by  step  through  the  materials  and  procedures.  Other  instructional  materials 
included  the  following: 

ACTA  Instructional  Software.  The  ACTA  Instructional  Software  consists  of  a  CD-ROM, 
which  contains  thorough  descriptions,  demonstrations,  and  practice  opportunities  for  each  of  the 
three  ACTA  interview  techniques.  The  software  also  contains  instruction  regarding  analysis  and 
application  of  data  collected  using  the  ACTA  techniques. 

ACTA  Instructional  Pamphlet.  The  ACTA  Instructional  Pamphlet  contains  an  outline  of  the 
ACTA  Instructional  Software,  incorporating  graphics  directly  from  the  software.  The  pamphlet 
served  as  a  guide  to  the  software  enabling  quick  reference  to  a  particular  area  of  interest.  See 
Appendix  D  for  a  copy  of  the  ACTA  Instructional  Pamphlet. 

ACTA  Job  Aids  (3).  Laminated  job  aids  listing  steps  for  each  knowledge  elicitation  technique, 
examples  of  knowledge  representation,  and  interview  tips  were  included  in  the  instructional 
package.  The  job  aids  could  be  referred  to  at  any  time  during  the  software  review.  Their  primary 
function  is  for  use  during  interviews  with  SMEs.  See  Appendix  D  for  copies  of  the  ACTA  Job 
Aids. 

Software  Reviewer  Background  Questionnaire.  The  Software  Reviewer  Background 
Questionnaire  was  administered  in  order  to  obtain  information  about  each  participant’s 
educational  and  professional  background.  The  Software  Reviewer  Background  Questionnaire  is 
included  in  Appendix  E. 

ACTA  Software  Review  Questionnaire.  Participants  were  asked  to  complete  a  four-page 
questionnaire  after  reviewing  the  software.  The  purpose  of  this  questionnaire  was  to  get  the 
participant’s  reactions  to  the  ACTA  Instructional  Software.  Participants  were  also  asked  to 
record  how  much  time  they  spent  in  each  of  the  three  modules  (i.e.,  Introduction,  Interview 
Techniques,  Applications).  The  Software  Review  Questionnaire  included  questions  in  which 
participants  were  asked  to  rate  the  ACTA  Instructional  Software  based  on  its  presentation  (e.g., 
The  concepts  were  clearly  presented.  The  information  was  well  organized),  and  open-end 
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questions  in  which  participants  were  asked  to  make  recommendations  for  improving  the 
software.  Other  questions  focused  on  the  usefulness  of  the  techniques  learned  and  the 
confidence  level  of  the  participant  to  conduct  interviews  with  SMEs  using  the  ACTA  interview 
techniques.  Response  options  ranged  from  a  “1”  (strongly  disagree)  to  a  “5”  (strongly  agree). 
See  Appendix  E  for  a  copy  of  the  ACTA  Software  Review  Questionnaire. 

Procedures.  Software  review  packages  were  mailed  to  all  of  the  participants.  Participants  could 
refer  to  the  software  review  instructions  to  guide  them  through  the  steps  of  the  evaluation.  In 
addition,  a  meeting  led  by  Dr.  Josephine  Ran  del  was  held  for  the  San  Diego  participants  to  go  over 
the  instructions  and  answer  any  questions  the  participants  might  have.  A  similar  session  was  held 
via  telephone  meetings  with  participants  in  other  parts  of  the  country.  For  the  first  step,  participants 
were  asked  to  review  the  ACTA  Instructional  Software.  They  were  informed  that  this  would  take 
approximately  1.5  to  2  hours  of  their  time.  After  reviewing  the  software,  participants  were  then 
instructed  to  fill  out  the  Software  Reviewer  Background  Questionnaire  and  the  ACTA  Software 
Review  Questionnaire.  When  the  Instructional  Designers  had  completed  the  software  review,  they 
were  instructed  to  mail  the  two  questionnaires  back  to  Klein  Associates.  All  of  the  participants  were 
encouraged  to  contact  Klein  Associates  researchers  throughout  the  software  review  to  ask  any 
questions  or  voice  any  concerns. 

Results 

Software  Review  Quantitative  Analysis.  Means  for  all  of  the  quantitative  questions  were 
calculated"  (see  Table  6).  These  data  show  that  in  working  through  the  software.  Instructional 
Designers  spent  most  of  their  time  in  the  Interview  Methods  module  (. M  =  59.00  minutes),  followed 
by  the  Introduction  module  (Af  =  18.29  minutes),  and  the  Applications  module  (M  =  15.86  minutes). 

For  each  of  the  closed-end  questions,  participants  reacted  positively  to  the  software.  For 
example,  participants  reported  that  using  the  ACTA  tools  to  conduct  an  interview  with  an  SME 
would  provide  them  with  important  information  about  the  cognitive  skills  involved  in  the  SME’s 
job  domain  ( M  =  4.43).  Overall,  participants  found  the  ACTA  Instructional  Software  to  be  very 
informative  (M  =  4.29).  In  addition,  the  Instructional  Designers  reported  that  the  concepts  were 
clearly  presented  (M  =  4.57)  and  the  information  was  well  organized  (M  =  4.57). 


Table  6 


Software  Review  Questionnaire  Means 


Mean 

Software  Review  Questions 

(Standard  Deviation) 

N=  7 

After  working  through  the  ACTA  instructional  software,  I  feel  confident  in  my  ability  to 

4.00 

use  the  ACTA  tools  to  conduct  an  interview  with  a  SME. 

(.58) 

Using  the  ACTA  tools  to  conduct  an  interview  with  a  SME  will  provide  me  with  the 

4.43 
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Table  6 

Software  Review  Questionnaire  Means 


Software  Review  Questions 

Mean 

(Standard  Deviation) 
JV  =  7 

After  working  through  the  ACTA  instructional  software,  I  feel  confident  in  my  ability  to 

4.00 

use  the  ACTA  tools  to  conduct  an  interview  with  a  SME. 

(.58) 

Using  the  ACTA  tools  to  conduct  an  interview  with  a  SME  will  provide  me  with  the 

4.43 

important  information  about  the  cognitive  skills  involved  in  his  or  her  job  domain. 

(.79) 

The  Cognitive  Demands  Table  will  be  a  useful  tool  to  help  me  organize  information  from 

4.00 

the  interviews  I  conduct  with  SMEs. 

(1.00) 

The  information  in  the  Cognitive  Demands  Table  will  facilitate  the  development  of  course 

4.29 

objectives. 

(.49) 

Overall,  I  found  the  ACTA  instructional  software  to  be  very  informative 

4.29 

(.76) 

The  ACTA  tools  described  in  the  software  are  very  similar  to  the  interview  techniques 

4.14 

that  I  typically  use  when  I  interview  SMEs. 

(.69) 

Given  your  current  position  within  your  organization,  rate  how  likely  you  would  be  to  use 

3.86 

the  Task  Diagram  in  an  interview  with  a  SME. 

(1.35) 

Given  your  current  position  within  your  organization,  rate  how  likely  you  would  be  to  use 

4.00 

the  Knowledge  Audit  in  an  interview  with  a  SME. 

(1.15) 

Given  your  current  position  within  your  organization,  rate  how  likely  you  would  be  to  use 

4.00 

the  Simulation  Interview  in  an  interview  with  a  SME. 

(1.00) 

The  concepts  were  clearly  presented. 

4.57 

(.53) 

The  information  was  well  organized. 

4.57 

(.53) 

The  examples  provided  were  helpful. 

Software  Review 
Questions. 

Note.  All  questions  used  a  5-point  scale  where  5  is  the  most  positive  and  1  is  the  least  positive.  Standard  deviations  are 
included  in  parentheses. 


Software  Review  Qualitative  Analysis.  Responses  to  open-ended  questions  were  generally 
positive.  One  participant  described  the  ACTA  Instructional  Package  as  “the  best  tutorial  I  have 
encountered  on  techniques  for  knowledge  acquisition  via  interview.”  Another  participant  said  that 
the  instructional  package  was  a  “good  start  on  a  difficult  concept  for  people  to  understand.”  Another 
participant  said,  “Techniques  are  all  applicable — so  much  of  our  training  is  in  cognitive  tasks.” 
Also,  several  participants  briefly  described  relevant  applications  for  the  ACTA  tools.  Some 
participants  offered  courses  for  which  the  ACTA  tools  would  be  relevant  (i.e.,  educational  technical 
classes),  while  other  participants  noted  jobs  where  the  ACTA  tools  would  be  relevant  (i.e.,  branch 
managers,  helicopter  pilots). 
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Requests  for  improvement  focused  on  better  instructions  for  getting  started  (i.e.,  “click  on  router 
to  get  started,”  “inform  user  that  audio  is  needed”).  Several  participants  also  expressed  a  desire  for 
additional  practice  opportunities  and  interactive  elements  in  the  software. 

Although  the  quantitative  data  indicate  that  participants  felt  confident  in  their  ability  to  use  the 
ACTA  tools  to  conduct  an  interview  with  an  SME,  responses  to  open-ended  questions  suggest  that 
many  participants  intend  to  spend  additional  time  reviewing  the  ACTA  instructional  materials. 
Others  reported  that  they  would  like  an  opportunity  to  practice  the  interview  techniques  with  a 
human  before  beginning  a  Cognitive  Task  Analysis.  These  comments  suggest  that  although  the 
instructional  package  clearly  presents  the  concepts  and  procedures  needed  to  conduct  ACTA,  many 
software  reviewers  believed  they  would  benefit  from  additional  preparation  time  and  practice 
opportunities. 

Navy  Instructional  Systems  Specialists.  As  the  ACTA  tools  were  designed  for  use  by  Navy 
ISSs,  we  also  examined  their  data  separately.  The  ISSs  responded  favorably  to  the  software.  All 
indicated  that  they  would  recommend  the  ACTA  software  to  colleagues.  They  all  offered  examples 
of  courses  they  believed  could  benefit  from  Cognitive  Task  Analysis.  One  ISS  commented  that, 
“Cognitive  Task  Analysis  is  a  major  deficiency  in  our  ISD  (instructional  system  design)  process.  It 
needs  to  be  done.” 

Suggestions  from  ISSs  about  improving  the  software  included  a  request  for  stronger  examples. 
One  reviewer  found  the  examples  or  tips  sections  in  the  software  required  a  lot  of  concentration. 
Another  ISS  found  navigation  difficult  and  asked  for  a  clearer  statement  of  the  purpose  of  the 
ACTA  tools.  One  ISS  found  the  graphics  harsh  and  the  background  aesthetically  unappealing. 

Workshop  vs.  Stand-Alone  Instruction.  Responses  to  the  question,  “After  working  through 
the  ACTA  Instructional  Software,  I  feel  confident  in  my  ability  to  use  the  ACTA  tools  to  conduct 
an  interview  with  an  SME,”  varied  little  from  people  who  learned  the  ACTA  techniques  via 
workshops  (M  =  4.18)  and  those  who  learned  the  ACTA  techniques  via  stand-alone  instruction  (M  = 
4-00). 12  However,  responses  to  open-ended  questions  indicate  that  people  who  learned  the  ACTA 
techniques  via  the  stand-alone  instruction  believed  they  would  benefit  from  additional  practice 
opportunities  such  as  those  offered  in  the  ACTA  workshops. 

Conclusions  and  Future  Research 

The  goal  of  this  project  was  to  begin  to  move  Cognitive  Task  Analysis  out  of  the  research 
community  into  applied  communities.  We  have  made  important  progress  in  this  direction.  We  have 
developed  streamlined  methods  of  Cognitive  Task  Analysis.  Our  evaluation  study  indicates  that  the 
methods  are  usable  and  aid  in  the  development  of  important,  accurate  training  materials  addressing 
cognitive  issues.  In  addition,  we  have  developed  a  CD-based  stand-alone  instructional  package 


^Statistical  analyses  indicate  these  means  do  not  differ  significantly,  (r<  1). 
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which  will  make  the  ACTA  tools  widely  accessible.  A  review  of  the  software  conducted  with  both 
Navy  ISSs  and  private  sector  Instructional  Designers  indicates  that  the  software  is  successful  in 
communicating  the  concepts  behind  the  ACTA  techniques  and  the  procedures  for  conducting  each 
technique.  The  ultimate  success  will  be  if  these  are  adopted  by  Navy  ISSs.  A  secondary  measure  of 
success  would  be  adoption  by  Instructional  Designers  in  the  private  sector. 

The  last  20-30  years  have  seen  an  exciting  cognitive  revolution  in  experimental  psychology. 
Our  hope  is  that  this  effort  can  help  bring  that  revolution  into  the  applied  community  in  the  form  of 
training  development  that  reflects  cognitive  as  well  as  behavioral  task  requirements. 

There  are  five  key  areas  in  which  future  research  could  expand  on  what  we  have  accomplished 
with  the  current  ACTA  project.  First,  there  is  a  need  for  team  Cognitive  Task  Analysis  methods. 
During  the  current  project  we  developed  the  Team  Schematic  to  capture  the  information  flow 
between  team  members.  Although  development  of  this  method  was  not  pursued  because  team 
Cognitive  Task  Analysis  methods  are  not  as  generally  applicable  as  individual  Cognitive  Task 
Analysis  methods,  the  need  for  such  tools  has  been  voiced  in  many  domains.  The  preliminary 
feedback  we  received  on  these  tools  was  generally  positive,  but  additional  research  needs  to  be 
conducted  to  revise  these  tools  and  to  develop  additional  methods  for  eliciting  the  knowledge  and 
skills  involved  in  team  decision  making  for  domains  in  which  information  flow  between  workers  is 
critical. 

Second,  ACTA  could  provide  valuable  information  for  systems  and  interface  design.  Thus  far, 
we  have  concentrated  our  efforts  on  developing  streamlined  Cognitive  Task  Analysis  methods  for 
use  by  training  professions  who  develop  instructional  materials.  The  expert  knowledge  and  skills 
elicited  using  the  ACTA  tools  also  have  implications  for  systems  and  interface  design.  Future 
research  could  develop  additional  tools,  or  modify  the  existing  ACTA  tools,  to  facilitate  their  use 
for  the  Decision-Centered  Design  of  systems  and  system  interfaces. 

Third,  improved  methods  for  designing  cognitive  skills  training  need  to  be  developed.  The 
ACTA  interview  techniques  produce  useful  data  that  describe  the  critical  judgment  and  decision¬ 
making  skills  involved  in  a  particular  domain.  Additional  research  is  needed  to  develop  methods  for 
translating  this  information  into  training.  How  does  a  training  developer  determine  effective  means 
to  train  novices  so  that  they  can  rapidly  acquire  these  skills?  For  example,  some  of  the  cognitive 
skills  may  be  trained  most  effectively  in  the  classroom,  whereas  other  skills  will  have  to  be  trained 
on-the-job.  Future  research  efforts  could  develop  new  tools,  or  modify  the  existing  ACTA  tools,  to 
facilitate  the  link  between  the  expertise  elicited  from  the  SME  and  the  optimal  techniques  for 
training  these  cognitive  skills  to  novices. 

In  addition,  future  research  should  investigate  whether  effective  knowledge  elicitation  and 
representation  tools  can  be  developed  that  reduce  the  role  of  the  interviewer  in  this  process. 
Although  the  ACTA  tools  are  the  most  streamlined  Cognitive  Task  Analysis  tools  available,  they 
still  require  a  significant  time  commitment  on  the  part  of  the  interviewer  and  SME.  It  typically  takes 
3  hours  to  use  all  three  interview  techniques  with  one  SME.  Future  research  could  explore 
alternative  techniques  in  which  the  SME  is  guided  through  a  “self-Cognitive  Task  Analysis” 
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process  by  using  a  structured  questionnaire  or  interactive  software.  Although  it  is  unlikely  that  this 
approach  would  result  in  as  rich  a  set  of  cognitive  demands  as  would  be  elicited  by  a  trained 
interviewer,  it  may  be  possible  to  identify  significant  cognitive  skills  with  this  type  of  method. 

A  final  direction  for  future  research  is  to  identify  better  metrics  to  assess  the  usability,  validity, 
and  reliability  of  Cognitive  Task  Analysis  tools  so  that  different  methodologies  can  be  compared.  It 
will  be  helpful  to  know  what  is  lost  using  streamlined  techniques,  and  also  what  is  gained.  It  would 
also  be  interesting  to  compare  some  of  the  more  extensive  Cognitive  Task  Analysis  techniques  to 
better  understand  the  strengths  and  weaknesses  of  each. 

This  report  has  described  a  streamlined  method  of  Cognitive  Task  Analysis,  called  ACTA — 
Applied  Cognitive  Task  Analysis — that  consists  of  three  interview  methods,  which  help 
practitioners  extract  and  represent  information  about  the  cognitive  demands  and  skills  required  for  a 
task  within  a  particular  domain.  Recent  changes  in  the  workplace  have  resulted  in  jobs  that  place 
increased  cognitive  demands  on  many  workers.  Knowledge  elicitation  and  representation  methods 
like  ACTA  will  allow  organizations  to  quickly  identify  the  key  cognitive  skills  involved  in 
particular  tasks  and  will  facilitate  the  development  of  cognitively-based  training  for  new  employees. 

The  current  effort  may  have  moved  the  field  two  steps  into  the  future  in  developing  streamlined 
Cognitive  Task  Analysis  techniques  and  creating  a  means  to  disseminate  the  ACTA  techniques  via 
stand-alone  instruction.  Cognitive  Task  Analysis  and  other  cognitive  technologies  will  never 
develop  into  widespread  use  as  long  as  they  are  designed  and  developed  solely  for  use  by  the 
research  community.  By  trying  to  make  these  methods  more  accessible,  we  hope  that  we  have 
contributed  to  the  growth  of  the  field  of  cognitive  instruction  and  cognitive  engineering. 
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Expertise  Literature: 

Characteristics  of  Expert  Performance 

It  is  important  to  consider  the  nature  of  expertise  if  one  is  to  conduct  a  successful  Cognitive 
Task  Analysis.  In  developing  Cognitive  Task  Analysis  techniques,  it  is  important  to  consider 
what  types  of  information  are  being  sought  and  how  they  are  likely  to  be  stored.  This  section 
contains  excerpts  from  the  expertise  literature  which  have  influenced  our  thinking  in  developing 
the  ACTA  techniques. 

Nature  of  expertise.  Shanteau  (1985)  has  distinguished  between  perceptual  expertise  (e.g., 
soil  judgments,  wine  tasting)  and  conceptual  expertise  (e.g.,  bridge  playing).  This  distinction 
seems  to  work  fairly  well.  If  a  domain  does  not  require  any  expertise,  then  it  is  not  an  appropriate 
subject  for  Cognitive  Task  Analysis.  That  is,  if  skill  does  not  continue  to  grow  for  a  period  of 
several  years,  sometimes  as  long  as  10  years,  sometimes  throughout  a  Subject  Matter  Expert’s 
(SMEs)  entire  career,  then  the  task  is  primarily  procedural  and  the  cost  of  Cognitive  Task 
Analysis  is  not  warranted. 

Expertise  is  domain  specific.  Experts  tend  to  be  experts  only  in  their  own  domains  of 
expertise.  The  reason  for  excellence  in  their  own  domains  is  vast  domain-specific  knowledge. 
This  has  been  found  in  studies  of  medical  diagnosis  (Johnson  et  al.,  1981),  and  taxi  drivers’  route 
knowledge  (Chase,  1983). 

Experts  perceive  patterns.  Experts  perceive  large  meaningful  patterns,  or  chunks  of 
information,  rather  than  individual  pieces  of  information.  This  is  proposed  to  be  a  reflection  on 
the  organization  of  the  knowledge  base  from  a  refined  sense  of  typicality  through  experience.  For 
example,  the  “perceptual”  advantage  in  this  view  is  proposed  not  to  reside  in  a  difference  in 
knowledge  base,  per  se,  between  the  expert  and  the  novice,  but  in  the  expert’s  ability  to  “see  the 
invisible,”  or  perceive  when  something  that  is  expected  to  be  there  is  missing. 

Experts  are  faster  and  make  fewer  errors.  Experts  are  also  proposed  to  be  able  to  perform 
more  efficiently,  consistently,  and  with  committing  fewer  errors  than  novice  performers.  This  is 
demonstrated  in  the  ability  of  chess  players  while  playing  “lightening  chess”  with  tight  time 
constraints  per  move  (Calderwood,  Klein,  &  Crandall,  1988).  This  ability  is  proposed  to  be  due 
to  the  skill  being  learned  to  automaticity,  where  the  expert  does  not  have  to  analyze  a  situation  to 
perform  well.  For  example,  while  driving  a  car  one  is  able  to  do  so  very  proficiently  with 
adequate  mental  resources  to  carry  on  a  conversation  and  listen  to  the  radio,  because  the  skill  of 
driving  is  automatic.  A  further  possible  reason  is  that  the  expert  does  not  have  to  conduct 
extensive  search  for  response  options  due  to  a  vast  library  of  learned  responses  to  typical 
conditions  (condition-action  rules). 

Experts  have  superior  memory  in  their  domain.  Another  finding  is  that  experts  have  superior 
long-  and  short-term  memory  in  their  areas  of  expertise  (Chase  &  Ericsson,  1982).  This 
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advantage  has  been  ascribed  to  various  mechanisms.  One  is  chunking.  The  information  is  said  to 
be  stored  in  meaningful  patterns  or  chunks  rather  than  in  terms  of  individual  pieces  of 
information.  This  means  that  the  expert  can  recall  more  than  the  novice.  Also,  the  chunks  of 
information  are  integrated  into  a  richer  more  meaningful  knowledge  base  which  aids  recall  in 
both  the  long-  and  short-term.  A  further  theory  puts  the  power  of  the  expert’s  memory  not  in  the 
head  of  the  expert  but  in  the  environmental  constraints  imposed  by  the  task  domain,  in  terms  of 
the  physical  environment  and  the  rules  that  govern  it.  The  expert  is  attuned  to  the  goal-relevant 
constraints  in  the  environment  and  thus  the  information  can  be  recalled  according  to,  or  with 
respect  to,  those  constraints  (Vicente  &  Wang,  in  press).  This  provides  a  mechanism  for  recall 
that  allows  the  expert  to  store  and  retrieve  the  information  in  meaningful  chunks  based  on  the 
rules  and  constraints  that  govern  the  task.  The  environmental  conditions  are  an  external  aid  to 
recall. 

Experts  see  and  represent  a  problem  at  a  deeper  level.  Novices  in  a  domain  tend  to  represent 
problems  at  a  more  superficial  level,  based  on  surface  features  of  a  problem,  and  on  learned 
rules,  as  opposed  to  thinking  about  a  problem  at  a  deeper  level.  Experience  provides  the  experts 
with  the  domain  knowledge,  patterns,  and  context  that  make  the  world  meaningful  to  them  at  a 
causal  level.  For  example,  when  asked  to  sort  physics  problems  into  categories,  novices  used 
literal  objects  stated  in  the  problem  description  to  sort  the  problems;  whereas,  experts  used 
principles  of  mechanics  to  organize  the  problems  (Chi,  Feltovich,  &  Glaser,  1981). 

Experts  spend  more  time  trying  to  understand  the  problem.  This  process  involves  building  a 
mental  representation  of  the  problem  from  which  relations  that  define  the  situation  can  be 
inferred.  Constraints  to  the  problem  can  then  be  added,  defining  the  boundaries  on  action.  This  is 
in  contrast  to  the  novice  who  jumps  right  in  and  begins  to  manipulate  the  surface  features  of  the 
problem.  An  understanding  of  the  problem  often  leads  to  a  more  efficient  path  to  the  solution 
without  being  sidetracked  by  irrelevant  information  and  without  following  fruitless  courses  of 
action.  The  importance  of  situation  awareness  and  problem  definition  is  a  central  theme  for 
understanding  expert  decision  making  in  real-world  contexts. 

Experts  have  strong  self-monitoring  skills.  Experts  have  an  ability  to  catch  themselves  when 
they  make  errors.  They  know  why  they  cannot  understand  a  problem,  and  they  know  when  to 
check  their  solution.  Novices,  on  the  other  hand,  do  not  have  the  ability  to  realize  their  own 
limitations,  and  do  not  have  a  refined  enough  sense  of  what  a  typical  solution  looks  like  to  be 
able  to  check  themselves  before  pursuing  a  false  line  of  action  or  reasoning. 

Experts  have  refined  perceptual  abilities.  This  perceptual  advantage  that  the  expert  has 
developed  exists  at  three  levels  according  to  Klein  and  Hoffman  (1993):  the  ability  to  see 
typicality,  the  ability  to  see  distinctions,  and  the  ability  to  see  antecedents  and  consequences 
(through  story  building  and  mental  simulation).  It  is  assumed  that  in  order  for  these  abilities  to 
work  for  the  expert,  an  extensive  knowledge  base  has  accrued  through  years  of  experience.  The 
assertion  is  not  that  the  less  experienced,  journeyman  performer  does  not  attempt  to  use  these 
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abilities,  it  is  more  that,  in  the  expert,  these  abilities  are  more  finely  attuned  based  on  a  greater 
number  of  past  experiences  on  which  to  draw. 

The  ability  to  see  typicality.  There  is  no  way  for  a  novice  to  judge  what  is  normal  and  what  is 
an  exception  (Chi,  Hutchinson,  &  Robin,  1988).  In  studies  with  expert  firefighters,  Klein, 
Calderwood,  and  Clinton-Cirocco  (1986)  found  that  the  rapid  size-up  of  a  situation  is  facilitated 
by  judging  typicality  of  the  situation  which  evokes  several  types  of  knowledge:  recognizing 
relevant  cues,  recognizing  events  that  are  expected,  recognizing  plausible  goals,  and  recognizing 
feasible  courses  of  action.  This  ability  allows  the  expert  decision  makers  to  direct  their  actions 
and  avoid  wasting  efforts  by  seeing  which  goals  are  feasible  given  the  situation.  They  are  able  to 
avoid  being  overwhelmed  by  a  flood  of  information  by  focusing  on  the  relevant  cues.  They  are 
able  to  recognize  when  a  situation  assessment  or  course  of  action  was  incorrect  based  on  their 
anticipation  of  expected  events  and  a  recognition  of  when  those  expectancies  are  violated.  They 
are  also  able  to  respond  rapidly  based  on  a  recognition  of  the  typical  course  of  action,  which  also 
means  that  experts  are  able  to  perform  proficiently  under  intense  time  pressure  (Calderwood, 
Klein,  &  Crandall,  1988). 

The  ability  to  see  distinctions.  Experts  are  particularly  good  at  making  fine  discriminations, 
compared  to  the  performance  of  the  novice  or  journeyman.  Good  examples  of  this  occur 
particularly  in  competitions  where  expert  judges  are  either  required  to  judge  performance  at  a 
task,  such  as  high  diving  or  gymnastics,  or  judge  some  attribute  of  a  particular  item,  such  as  prize 
bulls,  show  dogs,  or  watermelons.  The  inexperienced  audience  member  is  left  wondering  what  it 
was  about  the  winning  dive  A,  bull  X,  or  watermelon  Z  that  distinguished  them  from  the  rest  of 
the  competition.  This  ability  to  distinguish  subtle,  but  functionally  important  and  relevant, 
differences  between  two  or  more  similar  cues  or  pieces  of  information  is  a  skill  that  is  acquired 
through  active  participation  in  a  domain  for  many  years.  For  more  on  this  topic  the  reader  is 
referred  to  the  work  of  Shanteau  (Shanteau,  1985;  Phelps  &  Shanteau,  1978). 

The  ability  to  see  antecedents  and  consequences.  Experts  are  able  to  build  a  story  to  explain 
how  a  situation  arose  and  to  mentally  simulate  how  the  situation  will  develop,  or  how  a  course  of 
action  may  be  played  through.  This  ability  is  based  on  the  ability  to  perceive  typicality  and 
resemblances  to  many  past  experiences.  It  also  relies  on  the  subsequent  ability  to  generate 
expectancies  in  terms  of  what  will  happen  if  one  chooses  to  follow  a  particular  course  of  action. 
In  terms  of  playing  through  a  particular  course  of  action,  the  experts  may  predict  whether  the 
course  of  action  will  have  a  satisfactory  outcome.  If  the  outcome  is  not  satisfactory,  the  experts 
can  see  where  they  might  trip  up.  This  may  occur  because  of  a  realization  that  the  initial 
assessment  was  incorrect  in  some  way,  which  will  be  highlighted  by  the  mental  simulation,  or 
because  the  course  of  action  will  have  consequences  for  the  development  of  the  situation,  which 
may  call  for  an  alternate  subsequent  course  of  action. 

This  mental  simulation  of  a  course  of  action  has  been  likened  to  the  idea  of  “progressive 
deepening”  used  by  chess  masters  to  play  out  a  sequence  of  moves  and  the  opponents’  likely 
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responses  (de  Groot,  1946/1978).  For  more  about  the  functions  of  mental  simulation  in  skilled 
performance  see  Klein  and  Crandall  (1995). 

With  specific  reference  to  characteristics  of  expertise  in  decision-making,  Shanteau  (1987) 
presented  fourteen  characteristics  that  differentiated  the  decision  making  abilities  of  experts  from 
those  of  novices.  Briefly  these  include:  highly  developed  perceptual  and  attentional  abilities;  a 
sense  of  what  is  relevant  versus  irrelevant;  the  ability  to  simplify  complex  problems,  or  “make 
sense  out  of  chaos;”  a  superior  pattern-recognition  ability;  the  ability  to  communicate  their 
expertise  effectively;  an  ability  to  identify  and  react  to  exceptions  to  strategies;  a  strong  sense  of 
responsibility  in  backing  their  decisions;  an  ability  to  pick  solvable  problems;  strong  outward 
confidence;  a  strong  belief  in  their  ability;  adaptability  and  flexibility  in  their  response  to 
situations;  extensive  and  up-to-date  content  knowledge;  automaticity  in  cognitive  processing;  a 
greater  tolerance  to  stress;  creativity  in  finding  new  decision-making  strategies  when  presented 
with  atypical  situations;  and  an  inability  to  articulate  their  decision-making  processes. 

These  findings  have  been  found  to  be  robust  and  generalizable  across  many  varied  domains 
(Glaser,  1987).  The  fact  that  the  research  has  studied  individuals  within  a  rich  context,  and  that 
individuals  only  display  expertise  within  their  own  domains,  suggests  the  importance  of  the 
environmental  or  situational  factors  and  constraints  on  expertise. 
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AGENDA 

•  Overview  of  project 

•  Discussion  of  cognitive  skills  and  job 
performance 

•  General  interviewing  skills 

•  Description  of  learning  materials  to  be 
produced  by  participants 

•  Review  of  research  plan 
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PROJECT  OVERVIEW 
BACKGROUND  INFORMATION 
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•  SPONSOR:  The  Navy  Personnel 
Research  and  Development  Center 


•  PURPOSE:  To  evaluate  alternative 
interview  methods  for  use  by  Navy 
course  developers 


•  PRODUCT :  Improved  techniques  for 
the  development  of  course  materials  for 
training  Navy  personnel 
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PROJECT  OVERVIEW 
RESEARCH  PLAN 


•  Participants  will  attend  one  or  more 
workshops  to  develop  interviewing  skills 

•  Pairs  of  participants  will  conduct 
interviews  with  Electronic  Warfare 
Supervisors  (EWs) 

•  Each  participant  will  lead  one  interview 
with  an  EW  Supervisor 

•  Each  participant  will  observe  another 
interview  with  an  EW  Supervisor 

•  Interviews  will  be  conducted  at  Point 
Loma  or  Coronado 

•  Each  interview  is  scheduled  for  a  four- 
hour  block  of  time 


PROJECT  OVERVIEW  RESEARCH  PLAN  (continued) 

•  Participants  will  develop  instructional 
materials 

•  Each  participant  will  schedule  a  four-hour 
block  of  time  to  work 

•  Three  types  of  materials  will  be  developed 

—  a  cognitive  demands  table 

—  learning  objectives 

—  revisions  to  a  section  of  a  course 
manual 

•  Participants  will  also  be  asked  to  complete 
a  questionnaire  to  provide  us  with 
feedback  on  this  research  project 
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PROJECT  OVERVIEW 
BENEFITS  TO  PARTICIPANTS 


•  Participants  will  learn  about 
interviewing,  cognitive  skills,  and  how 
to  develop  course  materials 

•  Participants  will  be  paid  $250  for  their 
time 


IT  IS  ESSENTIAL  THAT  YOU  DO  NOT 
DISCUSS  THIS  PROJECT  WITH  OTHER 
PARTICIPANTS  UNTIL  THE  PROJECT  IS 
COMPLETED!! 
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COGNITION 


•  What  is  cognition? 

—  mental  activity 

—  what  goes  on  in  your  head 

•  Cognition  refers  to  the  processing  of 
information 

—  acquisition 

—  storage 

—  retrieval 

—  use 

—  transformation 

—  communication 
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COMPONENT  COGNITIVE 
PROCESSES 


•  ATTENTION :  mentally  focusing  on  a 
particular  stimulus 

•  PERCEPTION :  interpreting  sensory 
information  to  make  it  meaningful 

•  PATTERN  RECOGNITION:  classifying 
sensory  information  to  make  it 
meaningful 

•  MEMORY :  storage  and  retrieval 
processes 

•  HIGHER  ORDER  PROCESSES: 
judgments,  decision  making, 
problem  solving,  and  reasoning 
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COGNITIVE  SCIENCE 


•  Interdisciplinary  field  of  study  that 
includes 

—  psychology 

—  philosophy 

—  anthropology 

—  linguistics 

—  computer  science 

—  neuroscience 

•  Cognitive  scientists  assume  that 
thinking  involves  the  manipulation  of 
internal  representations  of  the  external 
world 

•  Cognitive  research  focuses  on  these 
internal  representations  — 

—  not  on  behaviors 

—  not  on  emotions 
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COGNITIVE  RESEARCH  METHODS 


•  Laboratory  methods 

—  Response  time 
—  Patterns  of  correct  or  incorrect 
responses 


•  Field  methods 

—  observations 

—  interviews 
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COGNITIVE  FACTORS  IN  JOB 
PERFORMANCE 


•  Traditional  approaches  to  the  development 
of  job  training  focused  on  behavioral  aspects 
of  the  job 

—  Researchers  decomposed  tasks  into 
component  procedures 
—  This  approach  neglects  the  cognitive 
factors  that  influence  job  performance 

•  Cognitive  Task  Analysis 

—  More  recent  approach  to  the 
development  of  training  materials 
—  Interviewers  identify  the  critical 
cognitive  factors  involved  in  a  task 
—  Training  materials  emphasize  the 
cognitive  as  well  as  the  behavioral 
aspects  of  the  job 
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CONDUCTING  INTERVIEWS  WITH 
ELECTRONIC  WARFARE 
SUPERVISORS 


•  Dress  appropriately 

•  Be  prepared  for  a  three-hour  interview 

—  Pencils/pens  and  paper 
—  Lots  of  questions 

•  Be  courteous  at  all  times 

•  Establish  rapport 

—  Make  sure  interviewee  is 
comfortable 

—  Introduce  yourself  and  explain 
why  you  are  there 

•  Thank  the  participant  for  his/her  help 
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ELECTRONIC  WARFARE  (EW) 
SUPERVISORS  TASK  DESCRIPTION 


•  You  will  be  interviewing  Electronic 
Warfare  Technicians  with  at  least  seven 
years  of  experience 

•  They  will  have  some  experience  as  an 
Electronic  Warfare  Supervisor 

—  They  have  supervised  other 
Electronic  Warfare  Technicians 
during  real-world  missions 
—  They  make  many  judgments  and 
decisions  while  serving  as  an  EW 
Supervisor 

•  For  the  purposes  of  this  project 

—  Focus  your  interview  questions  on 
the  cognitive  factors  involved  in  the 
EW’s  task  of  “signal  threat  analysis” 
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INSTRUCTIONAL  MATERIALS 
COGNITIVE  DEMANDS  TABLE 


•  Based  on  the  information  obtained  from 
your  interviews,  develop  a  table  that 
contains: 

—  Difficult  cognitive  demands  in  the 
job 

—  Why  these  demands  are  difficult 
—  The  important  cues  in  the 

environment  that  the  experienced 
EW  attends  to  in  this  situation 
—  What  strategies  the  experienced 
EW  Supervisor  uses  in  this  situation 
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COGNITIVE  DEMANDS  TABLE 
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Cognitive  Demands  Table  continued 
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INSTRUCTIONAL  MATERIALS 
COGNITIVE  DEMANDS  TABLE 


Purpose:  To  organize  ALL  the  information  you 

gathered  from  the  interviews  with  the 
two  EWs 

Remember:  The  focus  of  the  interviews  is  on  the 

cognitive  aspects  of  the  signal  threat 
analysis  task 


Common  questions: 

•  How  much  detail  should  you  include  in  the 
cognitive  demands  table? 

•  How  many  cognitive  demands  should  you 
identify? 
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INSTRUCTIONAL  MATERIALS 
LEARNING  OBJECTIVES 


•  Course  developers  specify  learning 
objectives  that  guide  their  design  of  course 
materials 

•  After  you  complete  your  interviews,  you  will 
be  asked  to  develop  learning  objectives 
based  on  the  cognitive  factors  you  have 
found  to  be  important 

•  A  learning  objective  is  a  statement  of  what 
the  trainee  can  do  after  training  (completing 
the  course  or  part  of  the  course) 

•  A  learning  objective  should  include  the 
following  two  elements: 

—  Behavior 
—  Conditions 


B-19 


INSTRUCTIONAL  MATERIALS 
LEARNING  OBJECTIVES 
BEHAVIOR  ELEMENT 


•  A  BEHAVIOR  usually  has  three 

elements 

—  the  subject  (which  is  always  the 
trainee  and  may  be  implied  or 
explicitly  stated) 

—  an  action  verb  describing  what  the 
trainee  is  expected  to  do 

—  an  object  describing  what  the  action 
verb  acts  upon 


EXAMPLE:  The  student  will  administer  the 
Standford-Binet  Intelligence  Test. 
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LEARNING  OBJECTIVES 
ACTION  VERBS 


PHYSICAL  SKILLS 

MENTAL 

SKILLS 

accomplish 

load 

achieve 

adjust 

locate 

analyze 

align 

manipulate 

calculate 

apply 

measure 

choose 

balance 

move 

compare 

calibrate 

operate 

compute 

change 

perform 

condense 

check 

plot 

decide 

clean 

position 

derive 

complete 

remove 

determine 

construct 

repair 

diagnose 

correct 

replace 

distinguish 

deenergize 

show 

evaluate 

demonstrate 

start 

interpret 

employ 

stop 

monitor 

energize 

test 

observe 

enter 

trace 

recognize 

exchange 

troubleshoot 

select 

inspect 

use 

solve 

install 

utilize 

synthesize 

KNOWLEDGE 

COMMUNICATION 


communicate 


define 


describe 


explain 


express 


identify 


illustrate 


list 


name 


state 


summarize 


tell 


write 


ADMINISTRATIVE 

SKILLS 


administer 


coordinate 


decide 


deliver 


draw 


fill  out 


instruct 


list 


manage 


report 


submit 
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INSTRUCTIONAL  MATERIALS 
LEARNING  OBJECTIVES 
CONDITION  ELEMENT 


•  The  CONDITION(S)  or  circumstances  under 

which  the  behavior  will  be  performed 

—  conditions  can  set  limits  or  restrictions  on 
performing  the  behavior 

—  conditions  can  describe  the  help  or 
assistance  given  the  trainee 

—  more  than  one  condition  can  apply  to  a 
learning  objective 

EXAMPLE  1 :  The  student  will  administer  the 
Standford-Binet  Intelligence  Test  to  children  without 
severe  emotional  problems. 

EXAMPLE  2:  With  his  or  her  supervisor’s  assistance, 
the  student  will  administer  the  Standford-Binet 
Intelligence  Test  to  children  with  severe  emotional 
problems. 
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INSTRUCTIONAL  MATERIALS 
LEARNING  OBJECTIVES 


•  The  learning  objectives  should  correspond  to 
the  cognitive  demands  listed  in  the  cognitive 
demands  table 

•  Typically,  participants  generate  at  least  as 
many  learning  objectives  as  cognitive 
demands  and  they  often  generate  more 
learning  objectives  than  cognitive  demands 

•  Do  not  combine  multiple  learning  objectives 
within  one  statement 
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INSTRUCTIONAL  MATERIALS 
REVISIONS  TO  COURSE  MANUAL 


•  After  you  have  completed  your 
interviews,  you  will  be  given  a  few 
pages  from  an  instructional  manual  for 
EW  Supervisors 

•  The  manual  describes  behavioral 
factors  that  are  important,  but  not  the 
cognitive  factors 

•  You  will  make  suggested  changes  or 
additions  to  the  manual  based  on  the 
cognitive  factors  you  have  identified  as 
being  important 

•  These  additions  to  the  manual  must  be 
written  in  complete  sentences 
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EXAMPLE 

Additions  to  existing  auto  repair  manual 


Tire  rotation 


1.  The  tires  should  be  rotated  at  the  specified  intervals  and  whenever  uneven  wear  is  noticed.  Since 
the  vehicle  will  be  raised  and  the  tires  removed  anyway,  check  the  brakes  (see  Section  28) 
at  this  time. 

It  is  a  good  idea  to  develop  a  habit  of  checking  your  tread  periodically,  so  that  you  can 
notice  changes  in  the  tires.  Other  indicators  of  irregular  tire  wear  occur  when  driving.  If 
you  notice  the  steering  wheel  pulling  to  one  side  or  shaking  as  you  drive,  you  should  check 
for  uneven  wear  on  the  tires.  Also,  if  you  hear  a  rhythmic,  slapping  sound  as  you  drive, 
uneven  tire  wear  is  likely  to  be  the  problem. 

When  visually  inspecting  the  tires  for  wear  patterns,  you  are  specifically  looking  for  places 
where  one  part  of  the  tire  is  worn  differently  from  the  rest  of  the  tire.  Some  examples  are 
provided  below.  Imagine  the  tire  going  down  the  road  in  order  to  visualize  what  must  be 
going  wrong  in  order  for  the  tire  to  wear  in  the  pattern  you  are  seeing. 

An  additional  diagnostic  procedure  is  to  manually  spin  the  tire  and  watch  the  rotation.  If 
you  see  a  wobble  as  the  tire  spins  or  it  appears  to  be  asymmetrical,  this  indicates  a  broken 
band  and  the  tire  should  be  replaced.  Also,  run  your  hand  along  the  sidewall  as  the  tire 
spins,  feeling  for  ridges.  Ridges  in  the  sidewall  also  indicate  a  broken  band.  Next,  run  your 
hand  along  the  tread  itself  If  you  feel  small  wavy  bumps,  a  front  end  alignment  is  needed. 

Finally,  grip  the  tire  and  aggressively  try  to  move  the  tire  sideways  and  up  and  down.  Some 
play  in  the  tire  is  to  be  expected.  If  you  can  easily  move  the  tire  sideways,  there  is  likely  to 
be  a  problem  with  the  ball  joint  or  steering  box,  or  a  loss  bearing.  Excessive  up  and  down 
movement  indicates  a  problem  with  the  steering  joint. 


2.  Radial  tires  must  be  rotated  in  a  specific  pattern  (see  illustration). 
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3.  Refer  to  the  information  in  Jacking  and  Towing  at  the  front  of  this  manual  for  the  proper 
procedures  to  follow  when  raising  the  vehicle  and  changing  a  tire.  If  the  brakes  are  to  be  checked, 
do  not  apply  the  parking  brake  as  stated.  Make  sure  the  tires  are  blocked  to  prevent  the  vehicle 
from  rolling. 

4.  Preferably,  the  entire  vehicle  should  be  raised  at  the  same  time.  This  can  be  done  on  a  hoist 
or  by  jacking  up  each  comer  and  then  lowering  the  vehicle  onto  jackstands  placed  under  the  frame 
rails.  Always  use  four  jackstands  and  make  sure  the  vehicle  is  firmly  supported. 

5.  After  rotation,  check  and  adjust  the  tire  pressures  as  necessary  and  be  sure  to  check  the  lug 
bolt  tightness. 

6.  Make  any  necessary  mechanical  repairs.  If  necessary,  have  an  alignment  done. 

7.  For  further  information  on  the  Wheels  and  Tires,  refer  to  Chapter  10. 
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Size-Up  Task  Description 


The  size-up  task  is  critical  to  each  incident  a  firefighter  confronts.  The  firefighter  must 
combine  preincident  information  (environmental  conditions,  water  supply,  area  knowledge, 
departmental  resources,  etc.)  with  what  s/he  sees  and  hears  from  the  time  of  dispatch  to  arrival  on 
the  scene.  This  information  is  combined  into  an  initial  assessment  or  size-up  of  the  situation,  and 
is  used  to  communicate  information  about  the  situation  to  other  firefighters  and  to  develop  a  plan 
for  a  course  of  action.  In  addition  to  the  initial  size-up,  information  is  gather  throughout  the 
incident  so  that  continual  updates  are  made  to  the  ongoing  size-up. 

We  believe  that  the  size-up  task  requires  considerable  cognitive  effort  and  skill.  Your  task 
is  to  interview  experienced  firefighters  about  this  task,  so  that  the  cognitive  elements  can  be 
documented  and  included  in  course  materials. 
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Signal  Threat  Analysis  Task  Description 

Combat  Information  Center  and 
the  Electronic  Warfare  (EW)Technician 

General  Background1 

Combat  Information  Center  (CIC)  is  the  technical  center  for  all  U.S.  Navy  ships.  This  room 
is  where  all  the  ship’s  sensors  (radars,  sonars,  radio  communications,  EW  equipment)  are 
controlled,  all  weapons  systems  (guns,  missiles,  torpedoes)  are  fired,  and  where  all  tactical 
decisions,  both  offensive  and  defensive,  are  made.  The  atmosphere  in  CIC  is  cold  and  dark  with 
only  blue  lights  and  the  sounds  of  computer  equipment,  and  radio  communications  nets  playing 
over  speakers;  there  are  no  windows  and  one  door.  CIC  is  wall-to-wall  consoles.  The  Tactical 
Action  Officer  (TAO)  is  in  charge.  He  is  the  officer  responsible  to  the  ship’s  Captain  (the 
Commanding  Officer  or  CO)  for  using  the  ship’s  weapons  in  combat.  If  you  have  seen  the  movie 
Hunt  for  Red  October,  you  know  what  it’s  like.  During  the  ship’s  highest  state  of  alert,  general 
quarters  (GQ),  all  consoles  will  be  used  (manned-up,  battle  stations).  When  GQ  is  set,  no  one 
leaves  his  position,  no  breaks,  no  quitting.  Everyone  is  serious  and  tensions  are  usually  high.  Most 
of  the  people  who  operate  the  consoles  (on  watch)  are  on  an  eight  hour  on  —  eight  hour  off 
schedule.  This  means  they  work  on  a  console  in  the  dark  for  16  hours  of  the  day,  and  most  never 
see  daylight  until  the  ship  reaches  port.  They  eat  —  stand  watch  —  sleep  —  stand  watch,  and  the 
cycle  continues.  This  is  where  EWs  spend  most  of  their  time  ...  on  watch  in  CIC. 

The  job  of  an  EW  in  CIC  is  to  find  enemy  radars  by  listening  to  the  sound  that  a  radar 
makes  when  it  scans  the  ship.  They  use  antennas  mounted  outside  the  ship  to  pick  up  these  radar 
signals.  The  main  objective  for  the  EW  is  to  keep  an  enemy  from  being  able  to  use  radar  to  target 
a  U.S.  Navy  ship  and  thus  fire  weapons  (missiles).  EWs  do  this  by  intercepting  radar  signals  that 
are  used  to  guide  missiles  to  targets.  If  they  know  the  enemy  is  using  radar  to  target  them,  they 
will  not  be  taken  by  surprise  by  an  enemy  attack.  If  missiles  are  launched,  they  can  counter  the 
missile  by  jamming  the  enemy  radar.  Jamming  the  enemy  missile,  or  enemy  ship’s  radar  is  also 
controlled  by  the  EW.  Jamming  can  be  accomplished  electronically,  or  by  launching  decoys  to 
fool  the  missile  or  enemy  radar  operator.  These  decoys  are  called  CHAFF.  They  provide  false 
targets  that  appear  on  radar  scans  and  confuse  the  missile  and  enemy  radar  operators  to  prevent 
them  from  successfully  targeting  U.S.  ships. 

Intercepting  the  enemy’s  radar  gives  another  advantage.  It  gives  the  EW  the  tactical 
location  of  the  enemy  or  the  enemy’s  bearing.  This  bearing  information  allows  the  launching  of 
U.S.  missiles  in  a  counter  attack  if  it  is  deemed  necessary  by  the  Captain  or  TAO.  If  an  attack  is 
not  made,  the  bearing  and  radar  (EW)  information,  along  with  the  ship’s  other  sensor  information, 
is  used  to  keep  an  overall  tactical  picture  (the  big  picture).  Then  senior  ship  personnel  will  try  to 
determine  the  enemy’s  intentions  and  develop  tactics  in  case  of  an  attack. 


'Summarized  from  notes  provided  by  EWI  Beltran,  FCTCPAC. 
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The  EW 


The  EW  operator  uses  the  equipment  to  actually  look  and  listen  for  the  enemy  radar  signals. 
Once  he  intercepts  a  signal  he  makes  an  initial  evaluation  of  the  signal  (friend  or  foe).  This 
evaluation  is  then  reported  to  the  EW  Supervisor,  along  with  the  bearing,  and  the  EW  operator 
moves  on  searching  for  other  signals.  The  EW  Supervisor  puts  these  data  into  his  console  and 
using  the  big  picture,  makes  the  ultimate  signal  evaluation  and  reports  this  to  other  stations  on  his 
ship  (like  the  TAO)  and  to  other  ships  in  the  area.  Once  a  signal  has  been  reported  to  the  other 
ships  it  is  referred  to  as  a  RACKET. 

If  the  signal  was  determined  to  be  an  immediate  threat  (Emergency  Racket)  the  Supervisor 
has  the  option  of  launching  decoys  —  in  the  case  of  an  enemy  missile  —  or  setting  GQ,  or  simply 
reporting  the  signal  and  monitoring  it  to  maintain  the  tactical  picture.  If  the  EW  operator 
misidentifies  a  signal,  and  the  EW  Supervisor  doesn’t  catch  the  mistake,  U.S.  ships  or  aircraft  may 
be  destroyed  by  enemy  missiles,  friendly  ships  or  aircraft  may  fire  at  other  friendly  units  (blue-on- 
blue  engagements),  or  U.S.  units  may  fire  at  the  wrong  target. 

In  May  1987  the  USS  Stark  was  hit  by  an  enemy  missile,  the  EWs  saw  the  enemy  targeting 
radar  and  heard  the  enemy  lock-on.  What  happened? 

In  April  1988  the  USS  Vincennes  fired  a  missile  and  destroyed  an  unarmed  airbus,  before 
the  captain  gave  the  order  to  fire  he  waited  as  long  as  he  could  for  an  EW  signal  —  none  was 
received. 


Electronic  Warfare 

EWs  as  a  rate  or  job  in  the  U.S.  Navy  perform  the  following:  Monitor  the  electromagnetic 
spectrum  with  the  purpose  of  ANTI-SHIP  MISSILE  DEFENSE  (ASMD).  EWs  operate  and 
maintain  the  ship’s  electronic  support  (ES)  and  electronic  attack  (EA)  equipment.  Watch  standing 
at  the  EW  OPERATOR  level  involves  operating  computer  equipment,  making  appropriate  voice 
reports,  and  keeping  a  written  log  of  intercepted  signals.  The  EW  SUPERVISOR  oversees  the  EW 
operations  and  makes  threat  analysis  decisions. 

The  step  between  EW  operator  and  EW  supervisor  is  usually  a  jump  of  5-8  years  of  job 
experience.  The  EW  rate  is  of  a  high  stress  nature.  Recently,  this  rate  was  opened  to  allow  females 
to  hold  this  position. 


Other  points  of  interest: 

—  EWs  support  both  combat  and  non-combat  roles  (EW-derived  information  can  be  used 
for  targeting  data  to  launch  an  attack,  although  their  primary  job  is  defensive). 
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—  EWs  maintain  and  launch  the  ship’s  only  SOFTKILL  weapons  systems  (electronic 
“jamming”  or  decoy  of  an  enemy  by  “chaff  j. 

—  EWs  operate  and  repair  their  own  equipment  (most  other  technical  rates  in  the  Navy 
include  two  separate  jobs,  one  to  operate  and  one  to  fix). 

—  EWs  must  have  access  to  classified  information  (most  of  the  job  is  classified  SECRET 
by  the  Department  of  Defense). 

—  The  written  logs  kept  by  EWs  on  watch  are  considered  official  documents  and  can  be 
used  in  legal  proceedings. 
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Terms 


Bearing:  Location  of  an  enemy  signal 

EA:  Electronic  Attack  —  actions  to  degrade  enemy  operations 

(jamming/decoys) 

Emitter:  An  enemy  signal 

EP:  Electronic  Protection  —  actions  taken  to  prevent  the  enemy  from  denying 

EWs  the  use  of  their  sensors  (don’t  let  enemy  jam  us). 

ES:  Electronic  Support  —  actions  in  support  of  friendly  operations 

Eval:  Evaluation  of  intercepted  signal 

Freq:  Frequency  of  an  intercepted  signal 

NSG:  NATO  Sea  Gnat  (a  chaff  decoy) 

Torch:  A  decoy  for  infra-red  (heat  seeking)  missiles 

Zulu  Time:  A  24-hour  clock  used  by  the  military  for  precise  timing  operations.  This 

avoids  confusions  from  different  time  zones.  Expressed  after  the  time,  (i.e., 
2300Z  is  the  same  as  1 1 :00  pm). 
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AGENDA 


•  Background  Information 

•  Task  Diagram  Interview 

•  Knowledge  Audit  Interview 

•  Simulation  Interview 

•  Summary 
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BACKGROUND 


•  Project  sponsored  by  Navy  Personnel 
Research  and  Development  Center 

•  Develop  ACTA  tools  for  use  by 
curriculum  developers 

•  Incorporate  cognitive  elements  of  the 
job  into  course  curriculum 
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Definition  of  Cognitive  Task  Analysis 


CTA  is: 

The  identification  of  the  cognitive  skills 
needed  to  perform  a  task  proficiently. 


•  Includes  interview  techniques 

•  Includes  methods  of  representing 
information 

•  Most  valuable  for  tasks  that  require 
cognitive  skills 

•  Focuses  on  key  decisions  and  how 
they  are  made 
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BEHAVIORAL  TASK  ANALYSIS 


•  The  attempt  to  decompose  a  task  into  its 
component  operations 

•  Strengths:  Flexible,  general,  logical, 
linked  to  training  interventions 

•  Limits:  Emphasis  on  procedures,  not  on 
judgments  and  decisions 
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Example  of  Behavioral  Task  Analysis 

2 . 1  Perform  N  ormal  T  akeoff  Operations 

2.1.2  Perform  Takeoff  Roll  Procedures 

2. 1 .2. 1  Position  aircraft  on  runway  centerline  and  stop 

2. 1 .2.2  If  required,  transfer  control  of  aircraft 

2. 1 .2.3  If  required,  comply  with  Standard  Policy  for 

Transfer  of  Aircraft  Control 

2. 1 .2.4  Select  HDG  HLD  on  MCP 

2 . 1 .2 . 5  Set  WX  radar  for  takeoff 

2. 1 .2.6  Release  brakes  and  set  takeoff  thrust 

2. 1 .2.7  Advance  power  on  both  engines 

2. 1.2.8  PF  call  for  EPR 

2. 1.2. 9  PNF  select  EPR  on  MCP 

2. 1.2.10  Engage  autothrottle  in  EPR  mode  as  engines  are 

accelerating  through  1 . 1  EPR 

2. 1 .2. 1 1  Comply  with  Standard  Policy  for  Takeoff  and 

Go/No  Go  Decision 

2. 1 .2. 1 2  Maintain  directional  control 

2. 1 .2. 1 3  Monitor  engine  and  flight  instruments 

2. 1 .2. 1 3 . 1  If  abnormality  exists,  Captain  decides 
and  initiates  rejected  takeoff 

2.1.2.14  Complete  standard  callouts  for  takeoff 

conditions 

2. 1 .2. 1 5  Comply  with  Standard  Policy  for  Takeoff  Flight 

Path  Control  Techniques 

2. 1 .2. 1 5. 1  Recognize  unstable  flight  path  condition 

and  be  prepared  to  execute  immediate 
recovery 
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Behavioral  Task  Analysis  of  Air  Traffic  Control 


I  Controller  scans  flight  progress  strips 


I 

_ ▼ _ 

j  Controller  mentally  projects  aircraft’s 
!  future  position 


f _ i _ 

Controller  monitors  aircraft  on 
|  display 


▼ 


!  Controller  views  aircraft  on  display 
j  and  mentally  projects  its  future 
j  position 


i 


Controller  updates 
flight  progress  strips 


Controller  inputs 
amendment  to 
computer 


▼ 

Repeats  process  with  another 
aircraft 


GOALS  OF  ACTA 


•  Provide  interview  format  that  allows  the 
interviewer  to  elicit  cognitive  elements  of 
the  job 

•  Provide  representation  format  that  presents 
the  data  in  a  readily  usable  form 

•  Can  be  incorporated  into  other  methods  to 
provide  a  more  complete  picture  of 
performance 

•  Easy  to  learn 

•  Easy  to  use 


i 
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ACTA  TOOLS 


•  TASK  DIAGRAM 

•  KNOWLEDGE  AUDIT 

•  SIMULATION 
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TASK  DIAGRAM 


•  Purpose  is  to  obtain  an  overview  of  the  task 


•  Purpose  is  to  identify  the  aspects  of  the  task 
requiring  expertise 


•  Purpose  is  to  frame  the  rest  of  the  CTA 
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Applied  Cognitive  Task  Analysis  Interview  Guide 


Go  into  this  interview  knowing  which  task  you  want  to  analyze.  You  will 
record  the  interviewee’s  responses  on  a  whiteboard  or  large  paper. 

Record  the  Task  of  Interest  at  the  top,  center  of  whiteboard. 

Ask  the  interviewee,  “Please  decompose  this  task  into  subtasks.  There  should 
be  at  least  three  sub-tasks,  but  no  more  than  six.” 

Record  each  Subtask  trom  left  to  right  across  the  whiteboard. 

Ask  the  interviewee,  “Which  subtasks  require  the  most  expertise?” 

Place  circles  around  the  tasks  that 
require  the  most  expertise  and 
squares  around  the  rest  of  the  tasks. 

Record  the  first  Subtask  that  requires 
expertise  on  the  whiteboard. 

Ask  the  interviewee,  “Please 
decompose  this  subtask  into  sub-sub 
tasks.  Again,  there  should  be  at  least  three,  but  no  more  than  six.” 

Record  the  Sub-sub  tasks  on  the  whiteboard. 

Ask  the  interviewee,  Which  of  these  sub  sub  tasks  require  the  most  expertise?” 

Circle  those  that  require  expertise  and  place  squares  around  the  rest. 

Continue  decomposing  subtasks  until  you  have  a  diagram  for  each  one  that 
requires  expertise.  DO  NOT  decompose  sub-sub  tasks. 

Use  this  diagram  when  conducting  the  Knowledge  Audit  to  limit  the  interview 
to  those  tasks  that  require  expertise. 


1995,  Klein  Associates  Inc. 
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TASK  DIAGRAM:  ROTATING  TIRES 


KNOWLEDGE  AUDIT 


•  Method  for  surveying  key  aspects  of 
expertise 


•  Contrasts  things  that  experts  know  and 
novices  don't  know 


•  Often  a  source  of  interesting  incidents 


CTA  Attempts  to  Identify  the  Cognitive  Skills 
Needed  to  Perform  a  Task  Well 


The  Past  and  the  Future 


Big  Picture 


Cognitive 
\  Aspects  / 
of 

Expertise 


Tricks  of  the  Trade 


Perceptual  Skills 


Recognition  of  Anomalies 


Improvising,  Noticing 
Opportunities 


Self  Monitoring  and  Adjustment 
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Applied  Cognitive  Task  Analysis  Interview  Guide 


m 


OWledge  Audit.  Contrasts  what  experts  know  and  novices  don’t. 


Preparation 


In  the  Task  Diagram  you  identified  the  sub-  and  sub-sub  tasks  that  require  the  most  expertise.  Go  into  this 
interview  knowing  the  sub-tasks  you  want  to  analyze.  _ _ __ 


Write  the  Task  of  Interest  at  top,  center  of 
whiteboard.  Divide  the  remaining  space  into  three 
columns  with  headings  that  match  the  illustration  on 
the  right. 


Example 
1.  Perceptual  Skills 
Example  of 
perceptual  skills 


Task  of  interest 


Why  Difficult  Cues  &  Strategies 


Expertise 

Perceptual 

Skills 

Anomaly 


Past  & 
Future 


Use  the  probes  listed  below  to  elicit  examples  of  the  Example  of  Anomaly 

various  aspects  of  expertise.  Record  the  first  example 
in  column  one.  Ask  questions  KA-3  and  KA-4  before  3._Past  &  Future 
moving  on  to  the  next  probe.  ^Example . . 


For  each  example,  ask,  “Why  is  this  task  hard  for  novices  or  why  don’t  novices  know  to  do  that?”  Record 
answers  in  middle  column  under  the  heading  Why  Difficult 

For  each  example,  ask,  “What  cues  or  strategies  do  you  use  in  this  situation?”  Record  answers  in  third  column 

under  Cues  and  Strategies. 

Knowledge  Audit  Probes 

Experts  delect  cues  and  patterns  and  make  discriminations  that  novices  can't  see.  Can  you  think  of  any  examples 
here? 

Experts  can  notice  when  something  unusual  happens.  They  can  quickly  detect  deviations.  They  also  notice  when 
something  that  should  happen  doesn’t.  Is  this  true  here?  Can  you  give  me  an  example? 

Experts  can  guess  how  the  current  situation  arose  and  they  can  anticipate  how  the  current  situation  will  evolve. 
Can  you  think  of  any  instance  in  which  this  happened,  cither  where  experts  were  successful  or  novices  fell  short? 


Big  Picture 

Tricks  of  the 
Trade 


If  you  were  watching  novices,  how  would  you  know  that  they  don't  have  the  big  picture? 
Arc  there  tricks  of  the  trade  that  you  use? 


I  Improvising 
or  Noticing 
Opportunities 

I  Self- 

monitoring  & 
Adjustment 


Can  you  recall  a  situation  when  you  noticed  that  following  the  standard  procedure  wouldn't  work?  What  did  you 
do?  Can  you  think  of  an  example  where  the  procedure  would  have  worked  but  you  saw  that  you  could  get  more 
from  the  situation  by  taking  a  different  action? 

Experts  notice  when  their  performance  is  sub-par,  and  can  often  figure  out  WHY  that  is  happening  (e.g.,  high 
workload,  fatigue,  boredom,  distraction)  in  order  to  make  adjustments.  Can  you  think  of  any  examples  where 


you  did  this? 


Optional  Probes 


Equipment 


Scenario 
from  Hell 


Unless  you  re  careful,  the  equipment  can  mislead  you.  Novices  usually  believe  whatever  the  equipment  says. 

Can  you  think  of  examples  where  you  had  to  rely  on  experience  to  avoid  being  fooled  by  the  equipment? 

If  you  were  going  to  give  someone  a  scenario  to  leach  someone  humilily-lhat  this  is  a  tough  job  -what  would  you 
put  into  that  scenario?  Did  you  ever  have  an  experience  that  taught  you  humility  in  performing  this  job  ? 


j 
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KNOWLEDGE  AUDIT  OF  AUTO  REPAIR 
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SIMULATION  INTERVIEW 


•  Highlight  cognitive  elements  of  the  task 
within  the  context  of  a  specific  incident 

•  Provide  access  to  an  expert's  thought 
processes 

•  Identify  potential  errors  a  novice  would 
be  likely  to  make 
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Applied  Cognitive  Task  Analysis  Interview  Guide 


Simulation  Interview.  Highlights  the  cognitive  elements  of  a  task. 

Preparation|  Obtain  a  simulation  of  the  task.  The  simulation  does  not  have  to  be  high  fidelity;  it  can  be  a 
paper  and  pencil  simulation,  video,  or  whatever  is  available. 

Ask  the  SME,  “Please  review  the  simulation  keeping  in  mind  that  I  will  be  asking  you  about 
the  decisions  and  judgments  you  would  have  made  in  this  situation.”  Offer  the  SME  pencil 
and  paper  on  which  to  keep  notes. 


Divide  a  whiteboard  into  6 
columns  with  headings  that 
match  the  illustration  on  the 


right. 


Events 
Decisions 
Judgm  ents 

Situation 

Assessment 

Actions 

Critical 

Cues 

Alternatives 

Potential 

Errors 

Event  # 1 

Event  #2 

Event  #3 

After  the  SME  has  reviewed 
the  simulation,  ask:  “Think  back  over  the  scenario.  Please  list  the  major 
events/judgments/decision  points  that  occurred  during  the  incident.  As  you  name  them,  I  am 
going  to  list  them  in  the  left  column  on  the  board.” 

For  each  event  in  the  left  column,  ask  the  questions  listed  below.  Ask  all  five  questions  about 
a  specific  event  before  moving  on  to  the  next  event.  Record  the  answers  to  each  question  in 
the  appropriate  column. 


■  Situation  What  do  you  think  is  going  on  here?  What  is  your  assessment  of  the  situation  at  this  point  in 
Assessment  time? 

■  Actions  What  actions,  if  any,  would  you  take  at  this  point  in  time? 

■  Critical  What  pieces  of  information  led  you  to  this  situation  assessment/action? 

Cues 

■Alternatives  Are  there  any  alternative  ways  you  could  interpret  this  situation?  Are  there  any  alternative 
courses  of  action  that  you  would  consider  at  this  point? 

■  Potential  What  errors  would  an  inexperienced  person  be  likely  to  make?  Are  there  cues  they  would 

Errors  miss? 
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This  appendix  contains  additional  information  about  the  evaluation  of  the  ACTA 
techniques  described  in  Section  V.  The  first  section  provides  additional  detail  about  the 
participants  in  the  study.  The  second  section  contains  the  definitions  of  the  coding  categories 
used  for  coding  cognitive  demands.  The  third  describes  additional  interviews  conducted  using  the 
Critical  Decision  method  and  explains  why  these  data  were  not  included  in  our  analyses.  The 
fourth  section  provides  an  in-depth  look  at  the  group  difference  data  and  a  discussion  of  the 
reasons  for  the  large  intra-group  variability  found  in  this  study. 


C.l  STUDY  PARTICIPANT  DEMOGRAPHICS 

Firefighter  Study  (Dayton) 

Participants:  Twelve  students  participated  in  the  study,  six  in  each  group.  Eight  female,  and  four 
male  participants  (two  males  in  each  group).  All  participants  were  recruited  from  graduate 
psychology  programs  in  the  Dayton,  OH  area.  The  schools  represented  were  University  of 
Dayton  and  Wright  State  University. 

Undergraduate  Degree:  Ten  of  the  12  participants  completed  their  undergraduate  degrees  since 
1993  (all  psychology  majors).  One  from  the  ACTA  group  completed  a  degree  in  Religious 
Studies  in  1986.  one  from  the  Unstructured  group  completed  a  degree  in  elementary  education  in 
1982. 

Graduate  Degree:  All  participants  were  in  a  graduate  psychology  program.  In  the  ACTA  group, 
there  were  three  first  year  students,  one  second  year,  and  two  third-year  (Ph.D.)  students.  In  the 
Unstructured  group  there  were  four  first-year  students  and  two  second-year  students. 

Work  Experience:  In  the  ACTA  group,  five  participants  had  had  one  year  or  less  of  work 
experience,  one  had  five  years  of  work  experience.  In  the  Unstructured  group,  three  had  one  year 
or  less  of  work  experience,  three  had  more  than  a  year’s  work  experience  (  four  years,  five  years, 
and  25  years). 

Related  Experience: 

Course  design:  In  the  ACTA  group,  one  person  had  some  course  design  experience,  the 
rest  had  none.  The  Unstructured  group  had  no  one  with  course  design  experience. 

Teaching:  In  the  ACTA  group,  three  people  had  teaching  experience  (graduate  teaching 
assistants);  in  the  Unstructured  group,  four  people  had  teaching  experience  (three  grad- 
TAs.  one  elementary  school  teacher). 

Writing  Learning  Objectives:  One  person  in  the  ACTA  group  had  had  some  experience. 
No  one  else  had  any  experience. 
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Instructional  System  Design:  No  one  had  any  experience  with  Instructional  System 
Design. 

Interviewing  Experience:  One  person  in  each  group  had  some  experience. 

Formal  Interviewing  Training:  Three  from  each  group  had  some  experience  (in  an 
Introduction  to  Clinical  Psych  class). 

Mean  time  to  complete  the  materials  development:  ACTA:  193.5  mins 

Non- ACTA:  210.67  mins 


Electronic  Warfare  Study  (San  Diego) 

Participants:  Eleven  people  participated  (five  in  the  ACTA  group,  six  in  the  Unstructured  group). 
Nine  females  and  two  males  (one  male  in  each  group).  All  students  were  from  psychology 
graduate  programs  in  the  San  Diego  area.  The  schools  included  California  School  of  Professional 
Psychology,  San  Diego  State  University,  and  University  of  California  at  San  Diego. 

Undergraduate  Degree:  Three  of  the  ACTA  group  received  their  degrees  since  1992,  one  in 
1985.  one  in  1989.  All  of  the  ACTA  group  held  undergraduate  degrees  majoring  in  psychology. 
In  the  Unstructured  group,  three  received  their  degrees  since  1990,  one  in  1984.  one  in  1987,  one 
in  1988.  Four  majored  in  psychology,  one  in  organizational  psychology,  one  in  social  ecology. 

Graduate  Degree:  In  the  ACTA  group,  two  students  were  in  their  fourth  year,  two  in  their  second 
year,  and  one  in  the  first  year  of  a  graduate  psychology  course.  In  the  Unstructured  group,  there 
were  three  first-year  students,  one  third-year,  and  two  fourth-year  students  in  psychology 
graduate  programs. 

Work  Experience:  Three  ACTA  group  participants  had  no  work  experience,  one  had  four  years, 
one  had  eleven  years.  Two  Unstructured  group  participants  had  no  work  experience,  the 
remainder  had  five,  nine,  ten,  and  thirteen  years  of  work  experience. 

Related  Experience: 

Course  Design:  One  participant  in  the  Unstructured  group  reported  any  experience  with 
course  design. 

Teaching  experience:  Two  ACTA  participants  had  graduate  Teaching  Assistant 
experience,  four  Unstructured  group  participants  had  graduate  Teaching  Assistant 
experience. 

Writing  Learning  Objectives:  No  one  had  any  experience. 
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Instructional  System  Design:  None. 


Interviewing  Experience:  Three  of  five  in  the  ACTA  group  had  had  various  interviewing 
experience.  Three  of  six  in  the  Unstructured  group  had  interviewing  experience. 

Formal  Interviewing  Training:  Three  ACTA  participants  had  some  formal  (mostly 
clinical)  interviewing  training.  Two  of  the  Unstructured  group  had  training  (clinical). 

Mean  time  to  complete  the  materials  development:  ACTA:  194.5  mins 

Non- ACTA:  216.17  mins 


C.2  DEFINITIONS  OF  COGNITIVE  CATEGORIES 
FOR  CODING  COGNITIVE  DEMANDS 

(adapted  from  Rasmussen  et  al.,  1994,  Cognitive  Systems  Engineering ,  p.  64) 

Identify  state  of  the  environment 

1)  Information  collection:  searching  for,  detecting,  and/or  recognizing  cues  or 
patterns  of  cues  in  the  environment  that  will  be  used  in  situation  analysis. 

2)  Situation  analysis:  interpreting  and/or  integrating  information  collected  from  the 
environment  (i.e.,  assessing,  determining). 

3)  Diagnosis:  explaining  the  current  state  of  the  environment  by  linking  observed 
events  to  causal  factors. 

Evaluation  of  current  state  relative  to  current  objective 

4)  Prediction:  consideration  of  future  conditions  relevant  to  the  current  situation. 

5)  Value  judgment:  consideration  of  personal  or  cultural  values  relevant  to  the 
current  situation. 

6)  Choice:  evaluating  (in  order  to  select  from)  a  set  of  two  or  more  alternative 
strategies  or  courses  of  actions  to  determine  the  "best”  one. 

Selection  of  course  of  action 

7)  Planning:  determining  the  set  of  actions  required  for  implementing  an  effective 
response  to  the  present  objective. 

8)  Scheduling:  determining  the  timing  for  implementation  of  planned  actions. 
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C.3  CRITICAL  DECISION  METHOD  INTERVIEWS 


Two  experienced  Klein  Associates  researchers  conducted  Critical  Decision  method 
interviews  with  three  experienced  Electronic  Warfare  (EW)  supervisors  in  San  Diego  in 
November.  1995  and  with  two  experienced  firefighters  in  Dayton  in  January,  1996.  The  Critical 
Decision  method  interviewers  attended  the  initial  workshop  given  for  students  in  the  firefighter 
study  in  October,  1995  to  learn  about  the  types  of  training  materials  they  would  be  required  to 
develop.  They  watched  the  videotape  of  the  EW  supervisor  used  in  the  initial  workshops 
provided  for  students  in  the  EW  study  prior  to  conducting  their  Critical  Decision  method 
interviews  with  EWs.  The  Critical  Decision  method  interviewers  spent  four  hours  developing 
Cognitive  Demands  Tables,  learning  objectives,  and  modifications  to  the  student  manual  based 
on  their  interviews  with  the  EWs.  They  spent  an  additional  four  hours  developing  Cognitive 
Demands  Tables,  learning  objectives,  and  modifications  to  the  student  manual  based  on  their 
interviews  with  firefighters. 

The  goal  of  gathering  Critical  Decision  method  data  was  to  have  a  set  of  criterion  data 
with  which  to  compare  data  gathered  using  other  techniques.  Unfortunately,  a  lack  of  available 
Subject  Matter  Experts  (SMEs)  prevented  the  completion  of  a  ftill-scale  Critical  Decision 
method  Cognitive  Task  Analysis.  A  typical  Cognitive  Task  Analysis  conducted  using  the  Critical 
Decision  method  includes  at  least  six  interviews  with  SMEs.  The  materials  generated  from  the 
Critical  Decision  method  interviews  with  two  (firefighter)  and  three  (EW)  SMEs  did  not  turn  out 
to  be  comprehensive,  and  thus  could  not  be  used  as  a  criterion  with  which  to  compare  the  data 
collected  using  other  techniques.  Qualitative  analysis  of  these  data  indicate  that  the  Cognitive 
Demands  Tables  generated  based  on  Critical  Decision  method  data  tended  to  include  higher- 
level,  broader-based  cognitive  demands  with  much  more  depth  to  fill  out  each  demand  than  the 
Cognitive  Demands  Tables  generated  based  on  ACTA  data  or  unstructured  interview  data. 

Although  it  would  be  interesting  to  explore  the  question  of  what  is  lost  using  streamlined 
Cognitive  Task  Analysis  techniques  as  opposed  to  an  in-depth  method  such  as  the  Critical 
Decision  method,  that  was  beyond  the  scope  of  this  evaluation  of  the  ACTA  tools. 


C.4  ADDITIONAL  GROUP  DIFFERENCE  DATA 

This  section  includes  differences  observed  in  the  questionnaire  data,  followed  by  a  table 
depicting  the  means  and  analyses  conducted  in  examining  the  validity  of  the  data  gathered  using 
the  ACTA  techniques  and  unstructured  interviews.  Interpretation  of  these  findings  is  not  included 
because  of  the  large  intra-group  differences  and  small  sample  sizes  in  our  data.  This  combination 
of  factors  makes  interpretation  of  group  differences  difficult. 

Interviewee  Questionnaire.  A  three-way,  mixed  design  ANOVA  taking  into  account  Domain 
(Firefighting,  EW),  Interview  Type  (ACTA.  Unstructured),  and  Question  (4  questions  from  the 
Interviewee  Questionnaire)  showed  significant  main  effects  for  interview  type  (F  (1,  20)  =  3.60,  e 
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=  .07)  and  question  (F  (3,  60)  =  6. 13,  p  =  .001).  The  main  effect  of  Question  is  of  little  interest  to 
this  study,  suggesting  that  the  responses  to  each  question  on  the  questionnaire  varied 
significantly.  The  main  effect  for  Interview  Type  indicates  that  the  experience  of  the  interviewee 
in  an  ACTA  interview  and  an  unstructured  interview  is  not  the  same. 

Interviewer  Questionnaire.  A  three  way,  mixed  design  ANOVA  taking  into  account  Domain 
(Firefighting,  EW),  Interview  Type  (ACTA.  Unstructured),  and  Question  (9  questions  from  the 
Interviewer  questionnaire)  showed  a  significant  main  effect  for  Question.  This  main  effect  is  of 
little  interest  in  this  study  as  it  merely  indicates  that  responses  varied  between  the  questions 
asked.  In  addition  to  this  main  effect,  a  three-way  interaction  between  Domain,  Interview  Type, 
and  Question  was  revealed. 

Question  8  demonstrated  a  significant  main  effect  for  Domain.  Participants  in  the  EW 
study  agreed  more  strongly  with  the  statement,  “I  was  able  to  use  the  information  to  make 
important  changes  in  the  course  materials,”  (M  =  3.82,  SD  =  .98)  than  participants  in  the 
Firefighter  study  (M  =  2.92,  SD  =  1 .08),  F  (I,  19)  =  4.84,  p  =  .04.  We  hypothesize  that  this 
difference  may  be  due  to  the  nature  of  the  materials  presented  to  the  two  groups  for  revision.  The 
sample  student  manual  used  in  the  Firefighter  study  contained  more  detail  than  the  sample 
manual  used  in  the  EW  study  due  to  differences  in  the  ways  in  which  the  two  skills  are  taught 
and  the  classification  level  of  much  of  the  EW  task.  This  provided  considerably  more  opportunity 
to  add  information  to  the  EW  manual. 

Table  7  depicts  the  groups  means  and  analyses  conducted  examining  group  differences 
for  each  of  the  validity  measures.  The  data  transformation  is  described  in  detail  in  Section  V 
of  this  report. 

As  discussed  in  Section  V.  we  attribute  the  lack  of  group  differences  in  this  study  to  large 
intra-group  differences.  In  examining  the  data,  the  large  standard  deviations  throughout  stand  out 
as  an  indicator  that  we  were  not  successful  in  reducing  intra-group  variability  sufficiently.  The 
difficulty  associated  with  interpreting  data  with  large  intra-group  variability  was  intensified  by 
the  small  sample  sizes  used  in  our  study.  Although  an  attempt  was  made  to  match  the  groups  on 
age.  gender,  and  education  level,  we  found  considerable  difference  in  the  students’  comfort  level 
and  ease  in  conducting  interviews  (as  observed  by  investigators). 
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Table  7 


Validity  Measures  and  Analyses  Exploring  Group  Differences 


Validity  Measure 

Firefighter 

Electronic  Warfare 

ACTA 

N=  6 

Unstructured 

N=6 

ACTA 

N=5 

Unstructured 

N=6 

1 )  Does  the  information  gathered 
address  cognitive  issues? 

Percent  of  items  in  Cognitive 
Demands  Tables  rated  as  cognitive. 

92% 

93% 

94% 

80% 

2)  Does  the  information  gathered  deal 
with  experience-based  knowledge  as 
opposed  to  declarative  knowledge? 

Proportion  of  information  in  the 
Cognitive  Demands  Table 
experienced  personnel  likely  to 
know,  averaged  across  users. 

.95 

(0.05) 

.93 

(0.03) 

.90 

(0.09) 

.79 

(0.18) 

Proportion  of  information  in  the 
Cognitive  Demands  Table  that  is 
relevant  for  a  Fireground 
Commander/Electronic  Warfare 
Supervisor,  averaged  across  users. 

.73 

(0.10) 

.88 

(0.07) 

.87 

(0.10) 

.93 

(0.07) 

3)  Do  the  instructional  materials 
generated  contain  accurate  information 
that  is  important  for  novices  to  learn? 

Mean  important  ratings  for 
modifications  to  the  student  manual. 
Ratings  on  a  3-point  scale,  where 

1  =  Not  important.  2  =  Somewhat 
important,  and  3  =  Very  important. 

2.06 

(.63) 

2.35 

(.42) 

2.50 

(.18) 

2.10 

(.44) 

Mean  important  ratings  for  learning 
objectives.  Ratings  on  a  3-point 
scale,  where  1  =  Not  important, 

2  =  Somewhat  important,  and 

3  =  Very  important. 

2.68 

(.21) 

2.67 

(.36) 

2.30 

(.21) 

2.44 

(.28) 

Table  7  (Continued) 


Validity  Measure 

Firefighter 

Electronic  Warfare 

ACTA 
N=  6 

Unstructured 

N=6 

ACTA 

N=5 

Unstructured 

N=6 

Mean  accuracy  ratings  for 

.90 

.90 

.12 

.16 

modifications  to  the  student  manual. 
Ratings  on  a  2-point  scale,  where 

0  =  Not  accurate.  1  =  Accurate. 

(.09) 

(.15) 

(.13) 

(.14) 

Mean  accuracy  ratings  for  learning 

.94 

.91 

.51 

.58 

objectives.  Ratings  on  a  2-point 
scale,  where  0  =  Not  accurate, 

1  =  Accurate. 

(.08) 

(.09) 

(.18) 

(.12) 

Note.  Standard  deviations  are  included  in  parentheses. 
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Knowledge  Audit  I  I  Simulation  Interview 
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Be  aware  of  the  energy  levels  of  all  participants  I  1  Be  prepared.  It  is  important  that  you  understand 
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One  of  ACTA’s  major  strengths  is  its  ability  to 
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Applied  Cognitive  Task  Analysis  Job  Aid 

1 U 

TASK  DIAGRAM 

Purpose:  The  Task  Diagram  is  intended  to  serve  as  a  road  map  to  the  rest  of  the  CTA.  The  Task  Diagram  acts 
as  an  advance  organizer,  providing  an  overview  of  the  task  and  identifying  the  cognitively  complex  elements  of 
the  task. 

How  to  get  started:  Before  you  begin,  have  clearly  in  mind  what  the  task  is  you  intend  to  investigate.  In  this 
interview,  you  want  to  find  out  about  the  interviewee’s  processes  as  s/he  performs  the  task  of  interest. 

CONDUCTING  THE  TASK  DIAGRAM  INTERVIEW 

□  Write  the  Task  of  Interest  at  top  of  whiteboard. 

□  Elicit  the  steps  required  to  do  the  task.  Record  them  across  the  board  from  left  to  right  in  chronological 
order.  Use  arrows  to  indicate  the  order  in  which  the  steps  occur.  (See  back) 

Ask  your  SME,  “ Think  about  what  you  do  when  you  (Task  of  Interest)  .  Can  you  break  this 

task  down  into  between  three  and  six  steps?" 

□  Elicit  information  regarding  which  of  the  steps  require  cognitive  skills.  Circle  the  elements  that  require 
cognitive  skills. 

Ask  your  SME,  "Of  the  steps  you  have  just  identified  which  require  difficult  cognitive  skills?  By 
cognitive  skills  I  mean  judgments,  assessments,  problem  solving— thinking  skills." 

At  this  point,  you  should  have  a  very  broad  overview  of  the  task,  with  an  indication  of  where  the  complex 
cognitive  skills  lie.  If  the  task  seems  too  big  or  the  steps  you  have  identified  are  too  broad  for  further 
investigation,  you  may  choose  to  focus  on  one  or  two  of  the  subtasks  you  have  identified  as  requiring  cognitive 
skills.  In  this  case,  you  should  complete  a  Task  Diagram  on  the  step(s)  you  have  chosen  to  focus  the  rest  of  the 
Cognitive  Task  Analysis. 


TIPS  FOR  DOING  THE  TASK  DIAGRAM  INTERVIEW 

□  Your  interviewee  may  immediately  start  talking  at  a  very  fine  level  of  detail.  Make  it  clear  early  on  that  you 
are  looking  for  a  very  broad  overview  with  this  interview.  You  will  be  interested  in  hearing  lots  of  stories 
and  details  later  in  the  session  (with  the  Knowledge  Audit  and  the  Simulation  Interview). 

□  If  your  interviewee  begins  listing  things  to  consider  rather  than  the  steps  of  the  task,  help  reframe  the  topic 
for  him/her.  “ What  do  you  do  when  you  (Task  ofjnterest )  ?" 

□  This  may  be  a  new  way  for  the  interviewee  to  think  about  the  job.  Give  him/her  time  to  think.  You  may 
need  to  repeat  or  rephrase  the  question. 

□  The  Task  Diagram  serves  as  a  road  map  to  the  rest  of  the  Cognitive  Task  Analysis.  You  are  not  trying  to 

elicit  detailed,  specific  cognitive  information  with  this  interview.  You  are  trying  to  get  a  sense  of  which 
parts  of  the  task  require  complex  cognitive  skills. _ * _ _ 
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EXAMPLE: 


Task  Diagram  of  Fireground  Commander's  Job  in  Commanding  Crew 


The  interviewer  decides  this  is  too  broad  —  really  wants  to  focus  on  the  assignment  of  tasks 
during  an  incident. 


EXAMPLE:  Task  Diagram  of  Assign  Tasks 


Assign 

overhaul 

crew 
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Applied  Cognitive  Task  Analysis  Job  Aid 

KNOWLEDGE  AUDIT 

Purpose:  The  Knowledge  Audit  provides  details  and  examples  of  cognitive  elements  of  expertise;  it  contrasts 
what  experts  know  and  novices  don’t. 

How  to  get  started:  You  used  the  Task  Diagram  to  identify  parts  of  the  job  that  require  skilled  judgment, 
decision  making  and  evaluation.  In  the  Knowledge  Audit  you  will  elicit  the  expertise  necessary  to  do  each  of 
those  tasks.  Use  the  Task  Diagram  to  help  you  decide 
which  tasks  and  subtasks  you  want  to  explore  with  the  I 

Knowledge  Audit.  Go  into  the  Knowledge  Audit  interview  Task  ofInterest 

knowing  what  you  want  to  analyze. 

CONDUCTING  THE  KNOWLEDGE  AUDIT  Example  Cues  &  Strategies  Why  Difficult? 

□  Write  the  Task  of  Interest  at  top,  center  of  whiteboard.  Past  &  Future 

Example... 

□  Divide  the  space  below  into  three  columns;  label  Big 

as  Shown.  Example— 


□  Elicit  an  example  of  one  element  of  expertise,  using  the  Example... 

definitions  and  probes  provided.  Start  with  the  first  ' - 

probe,  (e.g.,  “Is  there  a  time  when  you  walked  into  the 

middle  of  a  situation  and  knew  exactly  how  things  got  there  and  where  they  were  headed?) 

□  Elicit  information  for  the  remaining  two  columns  before  proceeding  to  another  element: 

—  Ask  your  SME,  “7n  this  situation,  how  would  you  know  this?  What  cues  and  strategies  are  you  relying 
on?”  Record  answers  in  middle  column  under  “Cues  and  Strategies.” 

—  Ask  your  SME,  “In  what  way  would  this  be  difficult  for  a  less-experienced  person ?”  What  makes  it  hard 
to  do?”  Record  answers  in  final  column  under  “Why  Difficult?” 

o  It  is  important  that  you  cover  the  six  basic  Knowledge  Audit  probes;  you  may  also  want  to  use  some  or  all  of 
the  optional  probes. 

TIPS  FOR  DOING  THE  KNOWLEDGE  AUDIT 

o  Examples  allow  you  to  get  at  specifics  and  help  you  understand  the  task  better.  Ask  for  an  example  for 
each  element  of  expertise. 

o  Don’t  try  to  write  everything;  but  write  enough  so  you  will  know  later  what  was  said  and  meant.  With 
practice  you  will  develop  a  sense  of  the  level  of  detail  you  need. 

□  Some  of  the  questions  may  take  a  few  minutes  for  the  SME  to  answer  thoughtfully;  don’t  rush;  give  the 
SME  time  to  think  over  what  you  are  asking  about. 

□  Confusion  about  what  to  write  and  in  which  columns  may  be  a  signal  that  your  SME  has  misunderstood 

your  question;  the  information  you  are  getting  is  not  what  you  expect.  You  may  want  to  take  a  timeout, 
restate  the  question,  and  check  that  your  SME  understands  what  you  are  trying  to  get  at. _ 


Task  of  Interest 

Example  Cues  &  Strategies  Why  Difficult? 

Past  &  Future 
Example... 

Big  Picture 
Example... 

Noticing 

Example... 
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ELICITING  INFORMATION  WITH  THE  KNOWLEDGE  AUDIT 

Provide  an  explanation  of  the  type  of  information  you  want;  then  ask  the  probe  questions.  You  can  read  the 
definitions  below  or  paraphrase  them. 

BASIC.  PROBES: 

□  Past  &  Future  Experts  can  figure  out  how  a  situation  developed,  and  they  can  think  into  the  future  to  see  where 

the  situation  is  going.  Among  other  things,  this  can  allow  experts  to  head  off  problems  before 
they  develop. 

Is  there  a  time  when  you  walked  into  the  middle  of  a  situation  and  knew  exactly  how  things 
got  there  and  where  they  were  headed? 

□  Big  Picture  Novices  may  only  see  bits  and  pieces.  Experts  are  able  to  quickly  build  an  understanding  of  the 

whole  situation — the  Big  Picture  view.  This  allows  the  expert  to  think  about  how  different 
elements  fit  together  and  affect  each  other. 

Can  you  give  me  an  example  of  what  is  important  about  the  Big  Picture  for  this  task?  What 
are  the  major  elements  you  have  to  know  and  keep  track  of? 

□  Noticing  Experts  are  able  to  detect  cues  and  see  meaningful  patterns  that  less-experienced  personnel  may 

miss  altogether. 

Have  you  had  experiences  where  part  of  a  situation  just  “ popped ’  out  at  you;  where  you 
noticed  things  going  on  that  others  didn  ’t  catch  ?  What  is  an  example  ? 

□  Job  Smarts  Experts  learn  how  to  combine  procedures  and  work  the  task  in  the  most  efficient  way  possible. 

They  don’t  cut  comers,  but  they  don’t  waste  time  and  resources  either. 

When  you  do  this  task,  are  there  ways  of  working  smart  or  accomplishing  more  with 
less — that  you  have  found  especially  useful? 

□  Opportunities/  Experts  are  comfortable  improvising — seeing  what  will  work  in  this  particular  situation;  they 
Improvising  are  able  to  shift  directions  to  take  advantage  of  opportunities. 

Can  you  think  of  an  example  when  you  have  improvised  in  this  task  or  noticed  an 
opportunity  to  do  something  better? 

Experts  are  aware  of  their  performance;  they  check  how  they  are  doing  and  make  adjustments. 
Experts  notice  when  their  performance  is  not  what  it  should  be  (this  could  be  due  to  stress, 
fatigue,  high  workload,  etc.)  and  are  able  to  adjust  so  that  the  job  gets  done. 

Can  you  think  of  a  time  when  you  realized  that  you  would  need  to  change  the  way  you  were 
performing  in  order  to  get  the  job  done? 

OPTIONAL  PROBES; 

□  Anomalies  Novices  don’t  know  what  is  typical,  so  they  have  a  hard  time  identifying  what  is  atypical. 

Experts  can  quickly  spot  unusual  events  and  detect  deviations.  And,  they  are  able  to  notice  when 
something  that  ought  to  happen,  doesn’t. 

Can  you  describe  an  instance  when  you  spotted  a  deviation  from  the  norm,  or  knew 
something  was  amiss? 

□  Equipment  Equipment  can  sometimes  mislead.  Novices  usually  believe  whatever  the  equipment  tells  them; 
Difficulties  they  don’t  know  when  to  be  skeptical. 

Have  there  been  times  when  the  equipment  pointed  in  one  direction,  but  your  own  judgment 
told  you  to  do  something  else?  Or  when  you  had  to  rely  on  experience  to  avoid  being  led 
astray  by  the  equipment? 


□  Self 

Monitoring 
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SIMULATION  INTERVIEW 


Applied  Cognitive  Task  Analysis  Job  Aid 


'  Purpose:  The  Simulation  Interview  provides  a  view  of  the  expert’s  problem  solving  processes  in  context.  This 
interview  will  provide  you  with  specific  detailed  information  about  an  expert’s  cognitive  processes. 

How  to  get  started:  You  will  need  to  obtain  a  simulation  of  the  task.  The  simulation  you  choose  should  address 
difficult,  challenging  elements  of  the  job.  It  does  not  have  to  be  high  fidelity;  it  can  be  a  paper  and  pencil 
simulation,  a  video  depicting  a  scenario,  or  whatever  is  available.  It  is  important  that  the  simulation  you  choose 
presents  a  challenging  scenario. 

CONDUCTING  THE  SIMULATION  INTERVIEW 

o  Divide  a  whiteboard  into  five  columns;  label  as  shown  on  the  back. 

□  Have  the  SME  experience  (i.e.  read,  watch,  interact  with)  the  simulation. 

—  Tell  the  SME,  “As1  you  experience  this  simulation,  imagine  you  are  the  (Job  you  are  investigating}  in 
the  incident.  Afterwards,  I  am  going  to  ask  you  a  series  of  questions  about  how  you  would  approach  this 
i  situation .” 

□  Elicit  a  list  of  the  major  events  in  the  simulated  incident  and  record  in  the  first  column. 

—  Ask  your  SME,  ‘‘‘‘Think  back  over  the  scenario.  Please  list  the  major  events  that  occurred  during  the 
incident.  These  events  could  include  judgments  or  decision  points.  As  you  name  them,  I  am  going  to  list 
them  in  the  left  column  of  the  board.'’ 

□  Begin  with  the  first  major  event  and  elicit  information  for  the  remaining  four  columns  before  proceeding  to 
the  next  major  event. 

—  Ask  your  SME,  “As  the  (Job  you  are  investigating )  in  this  scenario,  what  actions,  if  any,  would  you 
take  at  this  point  in  time?"  Record  answers  in  the  second  column  under  Actions. 

—  Ask  your  SME,  “ What  do  you  think  is  going  on  here?  What  is  your  assessment  of  the  situation  at  this 
point  in  time?"  Record  answers  in  the  third  column  under  Situation  Assessment. 

—  Ask  your  SME,  “ What  pieces  of  information  led  you  to  this  situation  assessment  and  these  actions?” 
Record  answers  in  the  fourth  column  under  Critical  Cues. 

—  Ask  your  SME,  “What  errors  would  an  inexperienced  person  be  likely  to  make  in  this  situation?" 
Record  answers  in  the  fifth  column  under  Potential  Errors. 
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TIPS  FOR  DOING  THE  SIMULATION  INTERVIEW 


□  Eliciting  major  events  is  critical  to  this  interview.  The  major  events  should  be  turning  points  or  segments  of 
the  story.  You  do  NOT  want  a  recount  of  the  entire  scenario. 

□  People  often  want  to  critique  the  simulation.  Assure  your  interviewee  that  you  are  interested  in  his/her 
critique,  but  that  for  the  first  part  of  the  interview,  you  would  like  to  work  with  the  scenario  as  it  has  been 
presented.  Be  sure  to  follow  up  and  ask  for  a  critique  at  the  end.  The  critique  can  yield  interesting  additional 
insight  and  is  worth  the  time. 

o  Don’t  try  to  write  everything;  but  write  enough  so  you  will  know  later  what  was  said  and  meant.  With 
practice  you  will  develop  a  sense  of  the  level  of  detail  you  need. 

□  Confusion  about  what  to  write  and  in  which  columns  may  be  a  signal  that  your  SME  has  misunderstood  your 
question;  the  information  you  are  getting  is  not  what  you  expect.  You  may  want  to  take  a  timeout,  restate  the 
question,  and  check  that  your  SME  understands  what  you  are  trying  to  get  at. 


EXAMPLE:  EXCERPT  FROM  A  FIREGROUND  SIMULATION 


Events 

Actions 

Situation  Assessment 

Critical  Cues 

Potential  Errors 

On-scene  arrival 

-  Account  for  people 
(names) 

-  Ask  neighbors  (but 
don’t  take  their 
word  for  it,  check  it 
out  yourself) 

-  Must  knock  on 
doors  or  knock  it  in 
to  make  sure  people 
aren’t  there 

-  It’s  a  cold  night, 
need  to  find  place 
for  people  who  have 
been  evacuated 

-  Night  (time) 

-  Cold — 15  degrees 

-  Dead  space 

-  Add  on  floor 

-  Poor  materials 
(wood)  punk  board 
metal  girders  — 
buckle  and  break 
under  fire 

-  Common  attic  in 
whole  building 

-  Not  keeping  track 
of  people  (could  be 
looking  for  people 
who  are  not  there) 

Initial  attack 

-  Watch  for  signs  of 
building  collapse 

-  If  signs  of  building 
collapse,  evacuate 
and  throw  water  on 
it  from  the  outside 

-  Faulty  construction: 
building  may 
collapse 

-  Signs  of  building 
collapse  include: 
what  walls  are 
doing,  cracks 
(building  ready  to 
collapse),  floor 
groans  (floor  ready 
to  cave  in),  metal 
girders  (click — 
coming  out  of 
wall — popping), 
cable  in  old 
buildings  holds  wall 
together,  fire 
collapses  walls 

-  Ventilating  the  attic, 
this  draws  the  fire 
up  and  spreads  it 
through  the  pipes 
and  electrical 
system 
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vim 


Thanks  for  participating  in  the  software  review  of  our  Applied  Cognitive  Task  Analysis 
(ACTA)  software.  Your  feedback  will  be  of  great  value  to  us. 


Software  Review 

•  Work  through  the  ACTA  software  to  learn  about  the  ACTA  techniques.  You  will 
learn  about  three  techniques  (the  Task  Diagram,  the  Knowledge  Audit,  and  the 
Simulation  Interview),  as  well  as  a  tool  to  help  you  analyze  data  collected  using 
the  ACTA  techniques  (the  Cognitive  Demands  Table).  We  predict  that  this  will 
take  1.5  to  2  hours  of  your  time. 

•  When  you  are  finished,  please  return  to  the  main  menu  before  exiting.  The 
system  will  tell  you  how  long  you  spent  in  each  module  of  the  software.  Please 
record  these  times  on  the  first  page  of  the  Software  Review  Questionnaire. 

These  times  will  give  us  a  realistic  measure  of  how  long  an  experienced 
professional  spends  using  the  software. 

•  Fill  out  the  Software  Review  Questionnaire  provided,  describing  your  reactions 
to  the  instructional  software  and  to  the  ACTA  techniques. 

•  Fill  out  the  Software  Reviewer  Background  form. 

Return  the  Software  Reviewer  Background  form  and  the  Software  Review  Questionnaire 
in  the  envelope  provided  no  later  than  March  21,  1997. 

Our  goal  is  to  have  all  the  software  review  data  collected  by  March  21, 1997.  If  it  looks  as 
if  this  will  not  work  with  your  schedule,  please  let  us  know  at  this  time. 
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SOFTWARE  REVIEWER  BACKGROUND  INFORMATION 

Please  answer  the  following  questions  so  we  can  get  a  better  understanding  of  your 
educational  and  professional  background. 

Name: 

Phone  Number: 

E-mail  address: 

Current  position  title: 

Number  of  years  at  this  position: 

Briefly  describe  your  previous  professional  experience  in  the  instructional  design  field: 

What  kinds  of  instructional  materials  do  you  typically  develop? 


Have  you  ever  conducted  a  Cognitive  Task  Analysis? 

Yes _  No _ 

If  yes,  what  methods  have  you  used  to  conduct  a  Cognitive  Task  Analysis? 


If  yes,  how  frequently  do  you  conduct  a  Cognitive  Task  Analysis? 


Briefly  describe  your  educational  background. 
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SOFTWARE  REVIEW  QUESTIONNAIRE 


Name, _ 

Date. _ 

Fill  out  this  questionnaire  AFTER  you  have  reviewed  the  ACTA  instructional  software. 


Thank  you  for  reviewing  the  ACTA  Instructional  Software.  Now  we  would  like  you  to 
give  us  your  reactions  to  the  software  by  answering  the  questions  below.  Before  you  answer  the 
questions,  please  record  the  amount  of  time  you  spend  in  each  module  of  the  software  in  the 
blanks  provided  below.  (If  you  returned  to  the  main  menu  before  exiting  from  the  program,  these 
times  should  be  displayed  on  the  screen;  if  you  exited  before  returning  to  the  main  menu,  just 
skip  to  the  Question  2.) 


1 .  Time  spent  viewing  the  different  modules  in  the  ACTA  Instructional  Software: 


Introduction 

minutes 

Interview  Methods 

minutes 

Applications 

minutes 

2.  Please  rate  the  extent  you  agree  or  disagree  with  the  following  statements  by  circling  a 
number  on  the  scale  below  each  statement. 


a.  After  working  through  the  ACTA  instructional  software,  I  feel  confident  in  my  ability  to 
use  the  ACTA  tools  to  conduct  an  interview  with  a  Subject  Matter  Expert  (SME). 


Strongly  Strongly 

Disagree  Agree 

1  2  3  4  5 
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b.  Using  the  ACTA  tools  to  conduct  an  interview  with  a  SME  will  provide  me  with 
important  information  about  the  cognitive  skills  involved  in  his  or  her  job  domain. 


Strongly 

Disagree 

1  2 


Strongly 

Agree 


c.  The  Cognitive  Demands  Table  will  be  a  useful  tool  to  help  me  organize  information  from 
the  interviews  I  conduct  with  SMEs. 


Strongly 

Disagree 

1  2 


Strongly 

Agree 


d.  The  information  in  the  Cognitive  Demands  Table  will  facilitate  the  development  of  course 
objectives. 


Strongly  Strongly 

Disagree  Agree 

1  2  3  4  5 


e.  Overall,  I  found  the  ACTA  instructional  software  to  be  very  informative. 


Strongly  Strongly 

Disagree  Agree 

1  2  3  4  5 


f.  The  ACTA  tools  described  in  the  software  are  very  similar  to  the  interview  techniques  that 
I  typically  use  when  I  interview  SMEs. 


Strongly 

Disagree 

1 


Strongly 

Agree 


2 


3 
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3.  Given  your  current  position  within  your  organization,  rate  how  likely  you  would  be  to  use 
each  of  the  ACTA  tools  in  an  interview  with  a  SME. 

a.  Task  Diagram 


Not  At  AH  Extremely 

Likely  Likely 

1  2  3  4  5 


b.  Knowledge  Audit 


Not  At  All 
Likely 

1  2 


Extremely 

Likely 


c.  Simulation  Interview 


Not  At  All 
Likely 

1  2 


Extremely 

Likely 


4.  Please  rate  the  extent  you  agree  or  disagree  with  the  following  statements  about  the  ACTA 
instructional  software. 

a.  The  concepts  were  clearly  presented. 


Strongly 

Strongly 

Disagree 

Agree 

1 

2 

3 

4 

5 

b.  The  information  was  well  organized. 

Strongly 

Disagree 

1  2  3 


Strongly 

Agree 
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c.  The  examples  provided  were  helpful. 


Strongly 

Disagree 

1  2 


Strongly 

Agree 


5.  What  suggestions  do  you  have  for  how  we  can  improve  the  ACTA  instructional  software? 


6.  Would  you  recommend  the  ACTA  instructional  software  to  other  colleagues? _ 

If  yes,  please  briefly  describe  the  relevant  applications  you  see  for  the  ACTA  tools. 


If  no,  why  not? 
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